Python: How to get hung process to release control back in concurrent.futures pool?

Question

I have a parallelized a large CPU-intensive data processing task using the concurrent.futures ProcessPoolExecutor method like shown below.

with concurrent.futures.ProcessPoolExecutor(max_workers=workers) as executor:
        futures_ocr = ([
            executor.submit(
                MyProcessor,
                folder
            ) for folder in sub_folders
        ])

        is_cancel = wait_for(futures_ocr)
        if is_cancel:
            print 'shutting down executor'
            executor.shutdown()

def wait_for(futures):
    """Handes the future tasks after completion"""

    cancelled = False

    try:
        for future in concurrent.futures.as_completed(futures, timeout=200):
            try:
                result = future.result()
                print 'successfully finished processing folder: ', result.source_folder_path

            except concurrent.futures.TimeoutError:
                print 'TimeoutError occured'


            except TypeError:
                print 'TypeError occured'



    except KeyboardInterrupt:
        print '****** cancelling... *******'
        cancelled = True
        for future in futures:
            future.cancel()

    return cancelled

There are certain folders where the process seems to be stuck for a long time, not because of some error in the code but due to the nature of the files being processed. So, I wanted to timeout those types of processes, so that they return if a certain time limit is exceeded. The Pool can then use the process for the next available task.

Adding the timeout in the as_completed() function gives an error while completing.

Traceback (most recent call last):
  File "call_ocr.py", line 96, in <module>
    main()
  File "call_ocr.py", line 42, in main
    is_cancel = wait_for(futures_ocr)
  File "call_ocr.py", line 59, in wait_for
    for future in concurrent.futures.as_completed(futures, timeout=200):
  File "/Users/saurav/.pyenv/versions/ocr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 216, in as_completed
    len(pending), len(fs)))
concurrent.futures._base.TimeoutError: 3 (of 3) futures unfinished

What am I doing wrong here, and what is the best way to cause timedout processes to stop and relinquish the process back to the Process pool?

score 0 · Answer 1 · answered Jul 02 '17 at 16:13

The concurrent.futures implementation does not support such use case.

The timeout which can be passed to its functions and methods allows to set for how long to wait for results but has no effect on the actual computation itself.

The pebble library supports such use case.

from concurrent.futures import TimeoutError
from pebble import ProcessPool

def function(n):
    return n

with ProcessPool() as pool:
    future = pool.schedule(function, args=[1], timeout=10)

    try:
        results = future.result()
    except TimeoutError as error:
        print("function took longer than %d seconds" % error.args[1])

Python: How to get hung process to release control back in concurrent.futures pool?

1 Answers1