Multiprocessing Process Pool Executor blocking submit function

Question

I am trying to turn my executor.submit function call into a blocking function so that it waits until the ProcessPoolExecutor pool has available workers.

In other words, initially it should print 1,2, then wait for 5 seconds, print 3,4, etc.

How can I achieve this?

import concurrent.futures
import multiprocessing
import time

def wait_f():
    time.sleep(5)
    return 1



if __name__ == '__main__':
    multiprocessing.freeze_support()
    global_results = []
    with concurrent.futures.ProcessPoolExecutor(max_workers=2) as executor:
        futures = []
        for j in range(10):
            future = executor.submit(wait_f)
            futures.append(future)
            print(j)
        for future in concurrent.futures.as_completed(futures):
            result = future.result()
            global_results.append(result)

score 1 · Accepted Answer · answered May 14 '23 at 18:06

Please do keep in mind that the Pool of workers pattern is designed to handle the concurrency for you so you don't have to. In other words, you should not worry about scheduling more tasks than workers as the Pool will already optimize the task flow for you.

This said, you can build from this answer a working solution.

from threading import Semaphore
from concurrent.futures import ProcessPoolExecutor

class TaskManager():
    def __init__(self, processes):
        self.workers = Semaphore(processes)
        self.executor = ProcessPoolExecutor(max_workers=processes)

    def new_task(self, function):
        """Start a new task, blocks if queue is full."""
        self.workers.acquire()
        future = self.executor.submit(function)
        future.add_done_callback(self.task_done)

    def task_done(self):
        """Called once task is done, releases the queue if blocked."""
        self.workers.release()


task_manager = TaskManager(2)
task_manager.new_task(wait_f)

Thanks for the answer! How could I keep track of the return values like I do here: `for future in concurrent.futures.as_completed(futures): result = future.result()` — Eric, May 15 '23 at 00:38
Just let `new_task` return the future object and use as you need. — noxdafox, May 15 '23 at 06:37

Multiprocessing Process Pool Executor blocking submit function

1 Answers1