The following code deadlocks instead of exiting when SystemExit
is raised in the asyncio task.
import asyncio
import multiprocessing as mp
def worker_main(pipe):
try:
print("Worker started")
pipe.recv()
finally:
print("Worker exiting.")
async def main():
_, other_end = mp.Pipe()
worker = mp.Process(target=worker_main, args=(other_end,))
worker.daemon = False # (default), True causes multiprocessing explicitly to terminate worker
worker.start()
await asyncio.sleep(2)
print("Main task raising SystemExit")
raise SystemExit
if __name__ == "__main__":
asyncio.get_event_loop().run_until_complete(main())
It prints just the following and then hangs:
Worker started
Main task raising SystemExit
If I then press Ctrl+C I get a traceback indicating it was waiting for p.join()
in multiprocessing.util._exit_function()
.
This is with Python 3.9.5 on Windows 10.
Superficially the reason seems to be as follows:
When SystemExit
is raised in the asyncio task it appears that all tasks are ended and then SystemExit
is re-raised outside the event loop. The multiprocessing library has registered multiprocessing.util._exit_function()
with atexit.register()
, and it gets called as the main process exits. This executes p.join()
on the worker process. Crucially, this happens before the pipe is closed, so it deadlocks with the worker waiting for the pipe to be closed (EOFError
) while the main process waits for the worker to exit.
The solution/workaround seems to be to make the worker a daemon process so that _exit_function()
will explicitly terminate it before p.join()
, which breaks the deadlock. The only problem with this is that it prevents the worker doing any clean-up before it exits.
The same problem does not occur in a non-asyncio application, and I'm not sure why it should be different. If the application is non-asyncio, then as the main process exits the pipe to the worker is broken and the worker exits as expected with an EOFError
. I've also confirmed that if the asyncio task is allowed to exit normally but then there is a raise SystemExit
after run_until_complete()
returns then it behaves as for the non-asyncio case - i.e. it exits properly.
Is this a bug in Python, or is it expected to behave this way?