4

I'm using a decorator for the thread pool executor:

from functools import wraps
from .bounded_pool_executor import BoundedThreadPoolExecutor

_DEFAULT_POOL = BoundedThreadPoolExecutor(max_workers=5)

def threadpool(f, executor=None):
    @wraps(f)
    def wrap(*args, **kwargs):
        return (executor or _DEFAULT_POOL).submit(f, *args, **kwargs)

where the BoundedThreadPoolExecutor is defined here

When I try to use the concurrent futures in a function decorated with @threadpool and then waiting all the futures withas_completed like

def get_results_as_completed(futures):
    # finished, pending = wait(futures, return_when=ALL_COMPLETED)
    futures_results = as_completed(futures)
    for f in futures_results:
        try:
            yield f.result()
        except:
            pass

for some worker defined like

from thread_support import threadpool
from time import sleep
from random import randint

@threadpool
def my_worker:
     res = {}
     # do something
     sleep(randint(1, 5))
     return res

if __name__ == "__main__":
   futures_results = get_results_as_completed(futures)
       for r in futures_results:
           results.append(r)

I cannot get the futures completed despite of the .result() call, thus resulting in a infinite loop on futures_results. Why?

loretoparisi
  • 15,724
  • 11
  • 102
  • 146
  • Where are you creating the futures? I don’t see any code filling the `futures` object you pass to `get_results_as_completed()`. I note that calls to `BoundedTreadPool.submit()` also *block* until the pool has a worker available, so by the time that a series of `my _worker` calls would complete there are *at most* 5 futures still incomplete. E.g. if you created 10 in a listcompehension then you’d have to wait 5x 1-5 seconds of sleep time before the list is produced. – Martijn Pieters Oct 21 '19 at 09:23
  • And your `get_results_as_completed()` will only ever complete once all futures in the input sequence are done, if you call it in a loop with the same sequence of futures you’d get the same results back, instantly. Try removing the blanket `except` or at the very least *log the exception thrown* to see if any of these have failed somehow. – Martijn Pieters Oct 21 '19 at 09:25
  • For now this isn’t compete enough to reproduce your issue, sorry. Note that there are two syntax errors in your last sample; `def my_worker:` is missing a parameter definition (if empty, use `()`) and the `for` loop in the `__main__` guard should not be indented further than the line before it. – Martijn Pieters Oct 21 '19 at 09:25
  • 1
    I just noticed: your `threadpool` decorator doesn't return the wrapper, so `my_worker` would be set to `None` instead. Presumably your real code has a `return wrap` line you dropped from your posted code. – Martijn Pieters Oct 21 '19 at 12:29
  • @MartijnPieters thank you it works now. Did not get why it should be off topics. Can you tell me please? – loretoparisi Oct 21 '19 at 13:07
  • 1
    The close banner tells you, as does my earlier comment. We can’t reproduce your problem because we don’t have sufficient information. – Martijn Pieters Oct 21 '19 at 18:28
  • @MartijnPieters updated to be more clear. – loretoparisi Oct 27 '19 at 13:41
  • Your code in the question still doesn't run, however. We can't reproduce your problem without a proper [mcve]. – Martijn Pieters Oct 27 '19 at 13:45

0 Answers0