0

I've encountered an error that I cannot explain while try to retrieve the results of futures submitted to process pool. I've stored the future objects in a list, and my best guess is that the future object reference is being deleted somehow so that list comprehension fails.

The error is at results = [j.result() for j in jobs] in async_jobs below. The traceback,

in <listcomp>
    results = [j.result() for j in jobs]
  File "lib/python3.6/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "lib/python3.6/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
IndexError: list index out of range

non-MVCE code

def _job(*args, **kwargs):
    """Does work with thread pool and returns True"""
    def _thread_job(*args,**kwargs):
        """Can define here because we are using threading and don't need to pickle"""
        ...
        return None

    with futures.ThreadPoolExecutor(max_workers=4) as t_executor:
        jobs = []
        for i in range(...):
            f = t_executor.submit(_thread_job, ..., ...)
            jobs.append(f)
        results = [j.results() for j in jobs]
    return True

def async_jobs():
    with futures.ProcessPoolExecutor(max_workers=8) as p_executor:
        jobs = []
        for i in range(...):
            f = p_executor.submit(_job, ..., ...)
            jobs.append(f)
        results = [j.result() for j in jobs]

if __name__=='__main__':
    async_jobs()
Josh Albert
  • 1,064
  • 13
  • 16
  • `IndexError: list index out of range` is probably thrown by the thread itself. That's the annoying thing with those jobs. When it crashes you just get the message. – Jean-François Fabre Feb 23 '18 at 13:37
  • 1
    I've tried your code (with small changes) and I get `f = p_executor(_job) TypeError: 'ProcessPoolExecutor' object is not callable`. I think you could create a [mcve] by replacing all the "..." by dummy values. – Jean-François Fabre Feb 23 '18 at 13:41
  • There is probably a typo in here: `results = [j.result() for j in jobs()]`. You call a list `jobs`. – Ilija Feb 23 '18 at 20:41
  • I'd have to agree with Jean and go further by saying it might be the issue with `_thread_job` function that is executed by threads. The way both pool executors work is to wrap exception in target function as future's exception. – Ilija Feb 23 '18 at 20:56
  • Typos fixed. It seems I'll have to include more exception handling. I did find an exception that occurs only when _thread_job is run in more than one thread. To remove all doubt, is it possible that there could be a problem with a subprocess performing multithreading? – Josh Albert Feb 26 '18 at 00:17

0 Answers0