Python: exec with ProcessPoolExecutor

Question

from concurrent.futures import ProcessPoolExecutor
import os
import time

def parInnerLoop(item):
    a = 2+item
    print(f'A. {a} Processing {os.getpid()} done on {item}\n')
    exec(open('mainWork.py').read())
    print(f'D. {a} Processing {os.getpid()} done on {item}\n')

def main():
    with ProcessPoolExecutor(max_workers=4) as executor:
        for itemNo in range(10):
            executor.submit(parInnerLoop, itemNo)
    print('done')

if __name__ == '__main__':
    main()

mainWork.py

print(f'B. {a} Processing {os.getpid()} done on {item}\n')
a = 12
print(f'C. {a} Processing {os.getpid()} done on {item}\n')

Error:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "D:\Program Files\Python\Python37\Lib\concurrent\futures\process.py", line 101, in _python_exit
    thread_wakeup.wakeup()
  File "D:\Program Files\Python\Python37\Lib\concurrent\futures\process.py", line 89, in wakeup
    self._writer.send_bytes(b"")
  File "D:\Program Files\Python\Python37\Lib\multiprocessing\connection.py", line 183, in send_bytes
    self._check_closed()
  File "D:\Program Files\Python\Python37\Lib\multiprocessing\connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

Problems:

The Error, it comes in the end of program.
I want to execute all code of mainWork.py as a part of parInnerLoop, such that change in value of a should be reflected in parInnerLoop as well as in mainWork.py and vice-versa. I got output as follows. The value of a does not change where the print starts with D. ..., I want it to be 12.

Output for item 0

A. 2 Processing 19784 done on 0
B. 2 Processing 19784 done on 0
C. 12 Processing 19784 done on 0
D. 2 Processing 19784 done on 0

What is going on? Any help please.

What I ultimately want to achieve: run same code in parallel with different settings, without interfering each other

Extra:

I tested ProcessPoolExecutor working properly as shown here.
I tested exec(open('run.py').read()) working as intended using following code:

main.py

def myFun():
    a = 1
    b = 2
    print(a)
    exec(open('run.py').read())
    #execfile("run.py")
    print(a)
    print(b)
    print(c)

def main():
    myFun()

if __name__ == '__main__':
    main()

run.py

a = a+5
b = 10
c = 15

output

I experienced the same problem with 3.7.6 on Linux. It appears to be an open bug — nodakai, Jul 24 '20 at 18:37
@nokada, I experienced the same problem with Python 3.7.8 on Windows. Looks like the but is not yet fixed. — Bernardo Trindade, Oct 19 '20 at 15:36

score 2 · Answer 1 · answered May 30 '21 at 10:13

Not a full answer, but it seems related to

Thank you for the report! After a quick investigation, it seems like the issue is because of recent changes in the concurrent.futures module. In short, they maintain a dictionary of weak references to some service threads (named "QueueManagerThread") and try to shut down whatever threads are in this dictionary when exiting a Python process. The dictionary is usually empty but if a reference to a thread exists this may cause an attempt of closing the thread which is not alive. In this case the original exception is raised.

https://youtrack.jetbrains.com/issue/PY-34432

Python: exec with ProcessPoolExecutor

1 Answers1