0

Python's multiprocessing Pool context manager exit calls .terminate(), but a very common pattern that does not use context manager is

p = Pool()
p.map(do)
p.close()
p.join()

I know that map blocks, and therefore when done in a context manager, there isn't a need to call .close() and .join() (or is it actually better to call them?):

with Pool() as p:
    p.map(do)

I also know that .close() and .join() are useful for non-blocking usages e.g. apply_async.

If there are some other important reasons to call .close() then .join(), why isn't it in the exit call of Pool context manager? If there are no other benefits, shouldn't we do the following instead:

p = Pool()
p.map(do)
p.terminate()
darkgbm
  • 335
  • 2
  • 11
  • 1
    If you are going to replicate the behavior of the context manager, why would you **not** use context manager? In your last example, all your are doing is not terminating the pool properly if there is an exception in the map. – Guimoute Mar 19 '23 at 20:39
  • @Guimoute I am not saying I won't use context manager; my question is: before context manager for Pool became available, **effectively** the common pattern is calling .close() then .join(), but after context manager became available, **effectively** the common pattern is calling .terminate(). Why the difference? Based on your comment on my last example, I see that one difference is if map has an exception, nothing following map would be called. But let's say we put whatever follows (close, join, terminate) all in the finally of a try ... finally. – darkgbm Mar 19 '23 at 21:12
  • `a very common pattern that does not use context manager is`, no it's not common and nobody uses it, you can call any function after `map` whether it is terminate or join or close or nothing at all, it won't matter as the pool is managed by the pool object and will be cleaned up in the destructor – Ahmed AEK Mar 20 '23 at 08:00
  • You only need to call terminate if you are in a long running function and want to terminate the pool yourself instead of waiting for the destructor, in which case a context manager is more suited anyway. – Ahmed AEK Mar 20 '23 at 08:09
  • @AhmedAEK Regardless of whether it is the right approach, it is not true that __nobody__ uses that pattern. A quick search on stack overflow and you have examples like https://stackoverflow.com/questions/44402085/multiprocessing-map-over-list-killing-processes-that-stall-above-timeout-limi?noredirect=1&lq=1, https://stackoverflow.com/questions/30222198/python-multiprocessing-pool-workers-hang-when-using-pool-map?rq=1. – darkgbm Mar 21 '23 at 02:33
  • since you bothered to open the source code, it is plain visible in there that there are _a lot_ of resources that have to be properly terminated and corner cases the `terminate` call takes care of, while `.close` and `.join` are far more simple. I´d say the `.terminate` call is an evolution which can handle more cases. – jsbueno Mar 28 '23 at 16:42

0 Answers0