how to add a coroutine to a running asyncio loop?

Question

How can one add a new coroutine to a running asyncio loop? Ie. one that is already executing a set of coroutines.

I guess as a workaround one could wait for existing coroutines to complete and then initialize a new loop (with the additional coroutine). But is there a better way?

Answer 1

To add a function to an already running event loop you can use:

asyncio.ensure_future(my_coro())

In my case I was using multithreading (threading) alongside asyncio and wanted to add a task to the event loop that was already running. For anyone else in the same situation, be sure to explicitly state the event loop (as one doesn't exist inside a Thread). i.e:

In global scope:

event_loop = asyncio.get_event_loop()

Then later, inside your Thread:

asyncio.ensure_future(my_coro(), loop=event_loop)

Answer 2

You can use create_task for scheduling new coroutines:

import asyncio

async def cor1():
    ...

async def cor2():
    ...

async def main(loop):
    await asyncio.sleep(0)
    t1 = loop.create_task(cor1())
    await cor2()
    await t1

loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))
loop.close()

Answer 3

If the task is to add a coroutine(s) to a loop that is already executing some coroutines, then you can use this solution of mine

import asyncio
import time
from threading import Thread

from random import randint


# first, we need a loop running in a parallel Thread
class AsyncLoopThread(Thread):
    def __init__(self):
        super().__init__(daemon=True)
        self.loop = asyncio.new_event_loop()

    def run(self):
        asyncio.set_event_loop(self.loop)
        self.loop.run_forever()


# example coroutine
async def coroutine(num, sec):
    await asyncio.sleep(sec)
    print(f'Coro {num} has finished')


if __name__ == '__main__':
    # init a loop in another Thread
    loop_handler = AsyncLoopThread()
    loop_handler.start()

    # adding first 5 coros
    for i in range(5):
        print(f'Add Coro {i} to the loop')
        asyncio.run_coroutine_threadsafe(coroutine(i, randint(3, 5)), loop_handler.loop)

    time.sleep(3)
    print('Adding 5 more coros')

    # adding 5 more coros
    for i in range(5, 10):
        print(f'Add Coro {i} to the loop')
        asyncio.run_coroutine_threadsafe(coroutine(i, randint(3, 5)), loop_handler.loop)

    # let them all finish
    time.sleep(60)
    

After execution of this example we will get this output:

Add Coro 0 to the loop
Add Coro 1 to the loop
Add Coro 2 to the loop
Add Coro 3 to the loop
Add Coro 4 to the loop
Coro 0 has finished
Adding 5 more coros
Add Coro 5 to the loop
Add Coro 6 to the loop
Add Coro 7 to the loop
Add Coro 8 to the loop
Add Coro 9 to the loop
Coro 1 has finished
Coro 3 has finished
Coro 2 has finished
Coro 4 has finished
Coro 9 has finished
Coro 5 has finished
Coro 7 has finished
Coro 6 has finished
Coro 8 has finished

Process finished with exit code 0

Answer 4

Your question is very close to "How to add function call to running program?"

When exactly you need to add new coroutine to event loop?

Let's see some examples. Here program that starts event loop with two coroutines parallely:

import asyncio
from random import randint


async def coro1():
    res = randint(0,3)
    await asyncio.sleep(res)
    print('coro1 finished with output {}'.format(res))
    return res

async def main():
    await asyncio.gather(
        coro1(),
        coro1()
    ) # here we have two coroutines running parallely

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Output:

coro1 finished with output 1
coro1 finished with output 2
[Finished in 2.2s]

May be you need to add some coroutines that would take results of coro1 and use it as soon as it's ready? In that case just create coroutine that await coro1 and use it's returning value:

import asyncio
from random import randint


async def coro1():
    res = randint(0,3)
    await asyncio.sleep(res)
    print('coro1 finished with output {}'.format(res))
    return res

async def coro2():
    res = await coro1()
    res = res * res
    await asyncio.sleep(res)
    print('coro2 finished with output {}'.format(res))
    return res

async def main():
    await asyncio.gather(
        coro2(),
        coro2()
    ) # here we have two coroutines running parallely

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Output:

coro1 finished with output 1
coro2 finished with output 1
coro1 finished with output 3
coro2 finished with output 9
[Finished in 12.2s]

Think about coroutines as about regular functions with specific syntax. You can start some set of functions to execute parallely (by asyncio.gather), you can start next function after first done, you can create new functions that call others.

Answer 5

None of the answers here seem to exactly answer the question. It is possible to add tasks to a running event loop by having a "parent" task do it for you. I'm not sure what the most pythonic way to make sure that parent doesn't end until it's children have all finished (assuming that's the behavior you want), but this does work.

import asyncio
import random


async def add_event(n):
    print('starting ' + str(n))
    await asyncio.sleep(n)
    print('ending ' + str(n))
    return n


async def main(loop):

    added_tasks = []

    delays = list(range(5))

    # shuffle to simulate unknown run times
    random.shuffle(delays)

    for n in delays:
        print('adding ' + str(n))
        task = loop.create_task(add_event(n))
        added_tasks.append(task)
        await asyncio.sleep(0)

    print('done adding tasks')

    results = await asyncio.gather(*added_tasks)
    print('done running tasks')

    return results


loop = asyncio.get_event_loop()
results = loop.run_until_complete(main(loop))
print(results)