3

I've written a Python library that currently makes several independent HTTP requests in serial. I'd like to parallelize these requests without altering how the library is called by users, or requiring users to be aware that calls are being made asynchronously under the hood. The library is meant for novice/intermediate Python users mostly using Jupyter, and I'd like it to work without introducing them to unfamiliar async/await semantics.

The following example, which works in Jupyter, illustrates what I'd like to achieve but requires use of await to invoke the code on the final line:

import asyncio

async def first_request():
    await asyncio.sleep(2)  # Simulate request time
    return "First request response"

async def second_request():
    await asyncio.sleep(2)
    return "Second request response"

async def make_requests_in_parallel():
    """Make requests in parallel and return the responses."""
    return await asyncio.gather(first_request(), second_request())

results = await make_requests_in_parallel()  # Undesirable use of `await`

I've found previous answers describing how to call async code from synchronous code using asyncio.run(). In the Jupyter example above, I can replace the final line with the following to create a working, importable Python module:

def main():
    """Make results available to async-naive users""" 
    return asyncio.run(make_requests_in_parallel())

results = main()  # No `await` needed to get results -- good!

This seems to be what I want. However, in Jupyter, the code will produce an error:

RuntimeError: asyncio.run() cannot be called from a running event loop

A comment on the same answer above explains that because Jupyter runs its own async event loop, there is no need (or, apparently, option) to start another one, so async code can "simply" be called using await. In my situation, though, avoiding await is why I wanted to use asyncio.run() in the first place.

This seems to suggest that existing synchronous libraries cannot, by any means, internally parallelize any operation using asyncio without altering their public API to require use of await. Is this true?

If so, are there more practical alternatives to asyncio that would let me parallelize a group of requests in an internal function without educating my users about async/await?

goodside
  • 4,429
  • 2
  • 22
  • 32

1 Answers1

2

I found a great solution for this: nest_asyncio.

Once installed, the working solution in Jupyter is as follows:

import asyncio
import nest_asyncio
nest_asyncio.apply()

async def first_request():
    await asyncio.sleep(2)  # Simulate request time
    return "First request response"


async def second_request():
    await asyncio.sleep(2)
    return "Second request response"


async def make_requests_in_parallel():
    """Make requests in parallel and return the responses."""
    return await asyncio.gather(first_request(), second_request())

def main():
    """Make results available to async-naive users""" 
    return asyncio.run(make_requests_in_parallel())

results = main()  # No `await` needed to get results
goodside
  • 4,429
  • 2
  • 22
  • 32