3

Is there a way to dynamically adjust the multiprocessing.Pool size? I'm writing a simple server processes that spawns workers to handle new tasks. Using multiprocessing.Process might be better suited (since the number of workers should not be fixed) for this case, but I need to execute a custom callback in my parent process after the spawned child has exited. For some reason, only mulitprocessing.Pool.apply_async seems to offer a callback parameter (multiprocessing.Process does not).

martineau
  • 119,623
  • 25
  • 170
  • 301
thyrel
  • 453
  • 1
  • 4
  • 13
  • Take a look at the answer to question [_multiprocessing - pool allocation_](http://stackoverflow.com/questions/10348156/multiprocessing-pool-allocation). – martineau Dec 07 '16 at 18:15

1 Answers1

1

Billiard processing Pool allows to grow and shrink the Pool size.

Pebble allows to run processes assigning a callback once done.

noxdafox
  • 14,439
  • 4
  • 33
  • 45
  • permalinks for billiard.Pool: [shrink](https://github.com/celery/billiard/blob/ff8efc5d689ef048f5203593390f1ff6a052a5d5/billiard/pool.py#L1285) and [grow](https://github.com/celery/billiard/blob/ff8efc5d689ef048f5203593390f1ff6a052a5d5/billiard/pool.py#L1297) – milahu Jun 27 '22 at 16:01