0

In Python ProcessPoolExecutor, do you need call shutdown after getting a BrokenProcessPool exception?

Say I have something like this:

pool = ProcessPoolExecutor(max_workers=1)

try:
    return pool.submit(
        do_something,
            ).result()
except concurrent.futures.process.BrokenProcessPool as pool_exc:
    pool = None
    return None
  • Is it a bad idea to just set pool = None do I need to call shutdown?
  • What args should I use with shutdown? pool.shutdown(wait=False, cancel_futures=True)?
cozos
  • 787
  • 10
  • 19
  • 1
    I'd say that depends on the context: If the pool is running other processes and you want to wait until they are done then you have to `.shutdown(wait=True)`. Otherwise the program is going forward and the processes still do thier thing. I'd rather use the `ProcessPoolExecutor` as a context manager. – Timus Mar 09 '23 at 13:22
  • If your "something like this" example is really what you are doing then you are *not* multiprocessing because you are submitting a task to a pool that can only run one task in parallel (because of the pool size) and your main process is immediately blocking until that task is completed so it is not doing any useful work in parallel with the process pool. – Booboo Mar 09 '23 at 18:49
  • The purpose of this is to sandbox execution inside the process. I am running C++ extensions using pybind and I don't want my application to crash due to a segfault. With the subprocess, my Python interpreter can survive abrupt process terminations. – cozos Mar 10 '23 at 02:04

1 Answers1

1

You don't have to call shutdown as that's the responsibility of the pool.

When the pool object is deallocated (set to None) another thread is going to terminate and join the workers, which is similar to calling shutdown(wait=True) from another thread so no work is cancelled, but since the pool is already broken, there should be no workers to join.

Even if the pool wasn't broken, most of the work will be done in another thread, so you don't have to explicitly call shutdown(wait=False) if you are sure there are no references to the pool anywhere in your code.

Ahmed AEK
  • 8,584
  • 2
  • 7
  • 23
  • Thank you Ahmed. For my understanding, if the pool object is deallocated while there are running processes in it, will they become "zombies"? How does the pool "destructor" work? Does the Python GC know that it needs to join the workers? – cozos Mar 10 '23 at 02:07
  • @cozos the implementation details of Cpython spawns another thread that checks if the `pool` is no-longer referenced anywhere (by a weak-ptr) in the other thread and calls `shutdown(wait=True)` from the other thread, this just instructs the processes to terminate after they finish all the work they have to do, they aren't killed, nor are tasks cancelled, but they will "eventually" terminate when they finish their tasks. – Ahmed AEK Mar 10 '23 at 08:02