8

I have Python 3.8 and cannot upgrade (I have dependencies that only work with Python 3.8), so this SO post does not answer my question. How do I prevent the OSError: handle is closed error? This happens when returning a result to the main thread. Here is a very trivial example:

from concurrent.futures import ProcessPoolExecutor


def my_func(num):
    return num + 1


def worker(data):
    with ProcessPoolExecutor(5) as executor:
        result = executor.map(my_func, data)
    return result


if __name__ == "__main__":
    arr = [1, 2, 3, 4, 5]
    print(arr)
    arr = worker(arr)
    print(tuple(arr))

This gives

[1, 2, 3, 4, 5]
(2, 3, 4, 5, 6)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\hendra11\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\process.py", line 102, in _python_exit
    thread_wakeup.wakeup()
  File "C:\Users\hendra11\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\process.py", line 90, in wakeup
    self._writer.send_bytes(b"")
  File "C:\Users\hendra11\AppData\Local\Programs\Python\Python38\lib\multiprocessing\connection.py", line 183, in send_bytes
    self._check_closed()
  File "C:\Users\hendra11\AppData\Local\Programs\Python\Python38\lib\multiprocessing\connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

Does anyone know what's causing this bug and how I can manually fix it?

If it matters, I'm on Windows 10 running Visual Studio Code.

UPDATE: This error goes away when I use ThreadPoolExecutor AND it is much faster, but I don't know why.

Will Da Silva
  • 6,386
  • 2
  • 27
  • 52
adam.hendry
  • 4,458
  • 5
  • 24
  • 51
  • 1
    For what it is worth, I do not see the error with 3.8.5 on windows 10 – JonSG Jun 23 '21 at 00:11
  • @JonSG Thanks for that. I'm using 3.8.7...honestly not sure what's happening. – adam.hendry Jun 23 '21 at 00:57
  • Similar issue here: https://github.com/python-adaptive/adaptive/issues/173 and the ticket at Python: https://bugs.python.org/issue36281 It's quite annoying, not solved for me :/ – Eric Burel Jul 05 '21 at 13:05
  • 1
    same to me, I've solved it as try/catch and ignore, since functionality works as needed but when you done with process it closes incorrectly with throwing an error. – akushyn Apr 26 '22 at 19:02

0 Answers0