How to reset an asyncio eventloop by a worker?

Question

I'm working with an asyncio forever() eventloop. Now I want to restart the loop (stop the loop and recreate a new loop) after a process or a signal or a change in a file, but I have some problems to do that:

Here are three simplified code snippets in which demonstrate some coroutine workers and a coroutine loop restarter:

#1st try:

import asyncio

async def coro_worker(proc):
    print(f'Worker: {proc} started.')
    while True:
        print(f'Worker: {proc} process.')
        await asyncio.sleep(proc)

async def reset_loop(loop):
    # Some process
    for i in range(5):  # Like a process.
        print(f'{i} counting for reset the eventloop.')
        await asyncio.sleep(1)

    main(loop)  # Expected close the current loop and start a new loop!

def main(previous_loop=None):
    offset = 0
    if previous_loop is not None:  # Trying for close the last loop if exist.
        offset = 1  # An offset to change the process name.
        for task in asyncio.Task.all_tasks():
            print('Cancel the tasks')  # Why it increase up?
            task.cancel()
            # task.clear()
            # task.close()
            # task.stop()

        print("Done cancelling tasks")
        asyncio.get_event_loop().stop()

    process = [1 + offset, 2 + offset]
    loop = asyncio.get_event_loop()
    futures = [loop.create_task(coro_worker(proc)) for proc in process]
    futures.append(loop.create_task(reset_loop(loop)))

    try:
        loop.run_forever()
    except KeyboardInterrupt:
        pass
    except asyncio.CancelledError:
        print('Tasks has been canceled')
        main()  # Recursively
    finally:
        print("Closing Loop")
        loop.close()
main()

Out[1]:

Worker: 1 started.
Worker: 1 process.
Worker: 2 started.
Worker: 2 process.
0 counting for reset the eventloop.
Worker: 1 process.
1 counting for reset the eventloop.
Worker: 2 process.
Worker: 1 process.
2 counting for reset the eventloop.
Worker: 1 process.
3 counting for reset the eventloop.
Worker: 2 process.
Worker: 1 process.
4 counting for reset the eventloop.
Worker: 1 process.
Cancel the tasks
Cancel the tasks
Cancel the tasks
Done cancelling tasks
Closing Loop
Closing Loop
Task exception was never retrieved
future: <Task cancelling coro=<reset_loop() done, defined at reset_asycio.py:11> exception=RuntimeError('Cannot close a running event loop',)>
Traceback (most recent call last):
  File "reset_asycio.py", line 40, in main
    loop.run_forever()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 425, in run_forever
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "reset_asycio.py", line 17, in reset_loop
    main(loop)  # Expected close the current loop and start a new loop!
  File "reset_asycio.py", line 48, in main
    loop.close()
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 63, in close
    super().close()
  File "/usr/lib/python3.6/asyncio/selector_events.py", line 96, in close
    raise RuntimeError("Cannot close a running event loop")
RuntimeError: Cannot close a running event loop
Task was destroyed but it is pending!
task: <Task pending coro=<reset_loop() running at reset_asycio.py:11>>
reset_asycio.py:51: RuntimeWarning: coroutine 'reset_loop' was never awaited
  main()
Task was destroyed but it is pending!
task: <Task pending coro=<coro_worker() running at reset_asycio.py:4>>
reset_asycio.py:51: RuntimeWarning: coroutine 'coro_worker' was never awaited
  main()
Task was destroyed but it is pending!
task: <Task pending coro=<coro_worker() running at reset_asycio.py:4>>
Task was destroyed but it is pending!
task: <Task pending coro=<coro_worker() running at reset_asycio.py:8> wait_for=<Future cancelled>>
Task was destroyed but it is pending!
task: <Task pending coro=<coro_worker() running at reset_asycio.py:8> wait_for=<Future cancelled>>

#2nd try:

.
.
.

def main(previous_loop=None):
    offset = 0
    if previous_loop is not None:  # Trying for close the last loop if exist.
        previous_loop.stop()
        previous_loop.close()
        offset = 1  # An offset to change the process name.

    process = [1 + offset, 2 + offset]
    loop = asyncio.get_event_loop()
    futures = [loop.create_task(coro_worker(proc)) for proc in process]
    futures.append(loop.create_task(reset_loop(loop)))

    try:
        loop.run_forever()
    except KeyboardInterrupt:
        pass
    except asyncio.CancelledError:
        print('Tasks has been canceled')
        main()  # Recursively
    finally:
        print("Closing Loop")
        loop.close()
main()

Out[2]:

Worker: 1 started.
Worker: 1 process.
Worker: 2 started.
Worker: 2 process.
0 counting for reset the eventloop.
Worker: 1 process.
1 counting for reset the eventloop.
Worker: 2 process.
Worker: 1 process.
2 counting for reset the eventloop.
Worker: 1 process.
3 counting for reset the eventloop.
Worker: 2 process.
Worker: 1 process.
4 counting for reset the eventloop.
Worker: 1 process.
Closing Loop
Task exception was never retrieved
future: <Task finished coro=<reset_loop() done, defined at reset_asycio.py:9> exception=RuntimeError('Cannot close a running event loop',)>
Traceback (most recent call last):
  File "reset_asycio.py", line 15, in reset_loop
    main(loop)  # Expected close the current loop and start new loop!
  File "reset_asycio.py", line 21, in main
    previous_loop.close()
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 63, in close
    super().close()
  File "/usr/lib/python3.6/asyncio/selector_events.py", line 96, in close
    raise RuntimeError("Cannot close a running event loop")
RuntimeError: Cannot close a running event loop
Task was destroyed but it is pending!
task: <Task pending coro=<coro_worker() done, defined at reset_asycio.py:3> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7efed846f138>()]>>
Task was destroyed but it is pending!
task: <Task pending coro=<coro_worker() done, defined at reset_asycio.py:3> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7efed846f048>()]>>

#3rd try:

.
.
.

def main(previous_loop=None):
    offset = 0
    if previous_loop is not None:  # Trying for close the last loop if exist.
        offset = 1  # An offset to change the process name.
        for task in asyncio.Task.all_tasks():
            print('Cancel the tasks')  # Why it increase up?
            task.cancel()

    process = [1 + offset, 2 + offset]
    loop = asyncio.get_event_loop()
    futures = [loop.create_task(coro_worker(proc)) for proc in process]
    futures.append(loop.create_task(reset_loop(loop)))

    try:
        loop.run_forever()
    except KeyboardInterrupt:
        pass
    except asyncio.CancelledError:
        print('Tasks has been canceled')
        main()  # Recursively
    finally:
        print("Closing Loop")
        loop.close()
main()

Out[3]:

Worker: 1 started.
Worker: 1 process.
Worker: 2 started.
Worker: 2 process.
0 counting for reset the eventloop.
Worker: 1 process.
1 counting for reset the eventloop.
Worker: 2 process.
Worker: 1 process.
2 counting for reset the eventloop.
Worker: 1 process.
3 counting for reset the eventloop.
Worker: 2 process.
Worker: 1 process.
4 counting for reset the eventloop.
Worker: 1 process.
Cancel the tasks
Cancel the tasks
Cancel the tasks
Closing Loop
Worker: 2 started.
Worker: 2 process.
Worker: 3 started.
Worker: 3 process.
0 counting for reset the eventloop.
1 counting for reset the eventloop.
Worker: 2 process.
2 counting for reset the eventloop.
Worker: 3 process.
3 counting for reset the eventloop.
Worker: 2 process.
4 counting for reset the eventloop.
Cancel the tasks
Cancel the tasks
Cancel the tasks
Cancel the tasks
Cancel the tasks
Cancel the tasks
Closing Loop
Worker: 2 started.
Worker: 2 process.
Worker: 3 started.
Worker: 3 process.
.
.
.

#Problem:

In the #3rd try, apparently I've done it, but print('Cancel the tasks') increases up after each restarting, what's the reason?!
Is there a better approach to overcome this problem?

Forgive me for the long question I tried to simplify it!

[NOTE]:

I'm not looking for the asyncio.timeout()
I also tried with another thread in order to restart the eventloop with an unsuccessful result.
I'm using Python 3.6

Why do you want to reset the event loop? That is an unusual idea. — Klaus D., Jan 01 '19 at 20:00
Can you explain what you are trying to achieve with this - i.e. what problem are you solving? — user4815162342, Jan 01 '19 at 21:39
@user4815162342 I'm trying to implement an SNMP collector with the several configurations which stored in a file, I parse these configs, each config is given to the coroutine SNMP collector (like `coro_worker()` in question) in init loop (`loop.create_task()`). my SNMP collector (`coro_worker()`) has an indefinite loop. Problem is that when the config file changes, I can't stop this event loop (`run_forever()`) and recreate the coroutine SNMP collector with the new config. — Benyamin Jafari, Jan 02 '19 at 05:46
This could be an XY problem. While you are focusing on how to reset the eventloop you should focus on how to properly end your task. — Klaus D., Jan 02 '19 at 06:18
@KlausD. Is it necessary to ensure the end of the tasks to stop or close the event loop? — Benyamin Jafari, Jan 02 '19 at 06:39
You are still asking the wrong question. You don't reset or stop the event loop except in very special cases (not yours). It usually ends when the program ends. — Klaus D., Jan 02 '19 at 07:46
@KlausD. My collector is not a finishable program, it is an infinite service, but it needs to restart and recreate the tasks with a signal or changing in the config file. I looking for to a mercilessly stop tasks method. If there is any other idea I will appreciate that. — Benyamin Jafari, Jan 02 '19 at 07:51
You can always stop the event loop with its `stop` method, then close it and start a new one. You can also cancel all your tasks and re-create them within the same event loop. — user4815162342, Jan 02 '19 at 08:15
@user4815162342 In the `#2 try` snippet code on my question I did it (`stop` then `close`) but I encountered with the mentioned error in the `Out[2]` in my question. I also tried with `cancel` task in `#3 try` snippet code, but I think the tasks remain in the background. — Benyamin Jafari, Jan 02 '19 at 08:30
Maybe you are overcomplicating things with the recursion and the stopping, etc. Take a look at [this code](https://pastebin.com/5p4dRxyc) - it monitors an external source (the file system) and, when a file is created, it just stops the loop. `main()` then has the code that cancels the tasks it has created, and replaces them with new ones. That looks like it should resolve your issue. — user4815162342, Jan 02 '19 at 10:32

user4815162342 · Accepted Answer · 2019-01-06T21:04:47.203

The recursive call to main() and the new event loop adds unnecessary complication. Here is a simpler prototype to play with - it monitors an external source (the file system) and, when a file is created, it just stops the loop. main() contains a loop that takes care of both (re-)creating and cancelling the tasks:

import os, asyncio, random

async def monitor():
    loop = asyncio.get_event_loop()
    while True:
        if os.path.exists('reset'):
            print('reset!')
            os.unlink('reset')
            loop.stop()
        await asyncio.sleep(1)

async def work(workid):
    while True:
        t = random.random()
        print(workid, 'sleeping for', t)
        await asyncio.sleep(t)

def main():
    loop = asyncio.get_event_loop()
    loop.create_task(monitor())
    offset = 0
    while True:
        workers = []
        workers.append(loop.create_task(work(offset + 1)))
        workers.append(loop.create_task(work(offset + 2)))
        workers.append(loop.create_task(work(offset + 3)))
        loop.run_forever()
        for t in workers:
            t.cancel()
        offset += 3

if __name__ == '__main__':
    main()

Another option would be to never even stop the event loop, but to simply trigger a reset event:

async def monitor(evt):
    while True:
        if os.path.exists('reset'):
            print('reset!')
            os.unlink('reset')
            evt.set()
        await asyncio.sleep(1)

In this design main() can be a coroutine:

async def main():
    loop = asyncio.get_event_loop()
    reset_evt = asyncio.Event()
    loop.create_task(monitor(reset_evt))
    offset = 0
    while True:
        workers = []
        workers.append(loop.create_task(work(offset + 1)))
        workers.append(loop.create_task(work(offset + 2)))
        workers.append(loop.create_task(work(offset + 3)))
        await reset_evt.wait()
        reset_evt.clear()
        for t in workers:
            t.cancel()
        offset += 3

if __name__ == '__main__':
    asyncio.run(main())
    # or asyncio.get_event_loop().run_until_complete(main())

Note that in both variants canceling the tasks is implemented by await raising a CancelledError exception. The task must not catch all exceptions using try: ... except: ... and, if it does so, needs to re-raise the exception.

Is the second section in your answer compatible with *Python 3.7*? — Benyamin Jafari, Jan 04 '19 at 16:10
@BenyaminJafari The second section is written specifically for Python 3.7 in mind. It can also work in Python 3.5/3.6, just replace the `asyncio.run` line with the commented-out line below it. — user4815162342, Jan 04 '19 at 23:02
Sometimes I encountered with this error in my workers (`work()`): `concurrent.futures._base.CancelledError` then the previous task runs with the new task after the change, simultaneously. — Benyamin Jafari, Jan 06 '19 at 07:52
@BenyaminJafari What do you mean by "encountered the error"? Are you perhaps catching all exceptions? Raising a `CancelledError` is how asyncio terminates a task. If your code is catching that exception, it should immediately re-raise it. — user4815162342, Jan 06 '19 at 08:48
My mean that I sometimes I confront with this error `concurrent.futures._base.CancelledError` in my `work()` method which contained with `try` and `exception`. in exception this error is shown, and previous task that I expected it remove remains with the new task with the new change. — Benyamin Jafari, Jan 06 '19 at 08:54
I am trying with a worker which contained with an `exception asyncio.CanclledError:` that raised immediately, but also in the worker placed a `finally`, is it dangerous? — Benyamin Jafari, Jan 06 '19 at 10:02
Problem fixed with a `raise` in `exception asyncio.CanclledError`. Thanks a lot. I'm going to edit your answer with this exception. I hope you don't mind. — Benyamin Jafari, Jan 06 '19 at 13:54
@BenyaminJafari The `raise` doesn't fit in the answer because that code doesn't catch exceptions in the first place. With your edit it catches `CancelledError` only to immediately re-raise it, which doesn't make sense and is unnecessary. (The code in the answer works correctly without the try/except/raise.) I've now added a paragraph that warns about accidentally catching `CancelledError`. — user4815162342, Jan 06 '19 at 21:07

How to reset an asyncio eventloop by a worker?

1 Answers1