1

I have a main function which I run with asyncio and then inside it I used event_loop.run_in_executor() to run mutliple processes of a blocking function.

What I wish to do is inside each of these processes to run a new asyncio loop for each one in order to execute async code.

So I have a main async function where I create multiple processes and I want to create a new asyncio loop in each of them. How can this be done?

KZiovas
  • 3,491
  • 3
  • 26
  • 47

1 Answers1

3

You can just call asyncio.run within your subprocess with a coroutine with the async work you want to do. A minimal example that spins up 5 processes each with their own event loops:

import asyncio
from concurrent.futures import ProcessPoolExecutor


def run_loop_in_process():
    async def subprocess_async_work():
        await asyncio.sleep(5) #whatever async code you need
    asyncio.run(subprocess_async_work())

async def main():
    loop = asyncio.get_running_loop()
    pool = ProcessPoolExecutor()
    tasks = [loop.run_in_executor(pool, run_loop_in_process) for _ in range(5)]
    await asyncio.gather(*tasks)

if __name__ == "__main__":
    asyncio.run(main())
Matt Fowler
  • 2,563
  • 2
  • 15
  • 18
  • so you can call asyncio.run() within asyncio.run() ? – KZiovas Oct 26 '21 at 20:46
  • 1
    @KZiovas yes, since you're calling it within a separate subprocess you'll get a new event loop so it is safe. Do keep in mind that two event loops in the same process will cause problems. – Matt Fowler Oct 26 '21 at 20:53
  • i tested it does work, one thing I had to change was various object `self.variable` that I was reusing in the functions that I wanted to run in the sub-processes. I had to re-declare the variables in each sub-process because they were trying to use the same resources (some client objects). But that is to be expected. – KZiovas Oct 26 '21 at 22:22