0

I am using executor.map and storing results in 'results'.

Following code works without any problem on Jupiter notebook. However, it crashes when executing the python script in Windows command prompt.

rop_test.py

print ('start of process')
import concurrent.futures
from funcfile import func
if __name__ == '__main__': 
    with concurrent.futures.ProcessPoolExecutor() as executor:
        results = executor.map(func, varlist1,varlist2)
    global master_list
    master_list=list(results)

print(master_list)

#do something with master_list

funcfile.py

def func(var1,var2):
    var3=var1+var2
    return var3

Error

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Administrator\notebooks\rop_test.py", line 640, in <module>
    print (master_list)
NameError: name 'master_list' is not defined
Finished in 0.0 second(s)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Administrator\notebooks\rop_test.py", line 640, in <module>
    print (master_list)
NameError: name 'master_list' is not defined
Traceback (most recent call last):
  File "rop_test.py", line 634, in <module>
    master_list=list(results)
  File "C:\ProgramData\Anaconda3\lib\concurrent\futures\process.py", line 483, in _chain_from_iterable_of_lists
    for element in iterable:
  File "C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py", line 598, in result_iterator
    yield fs.pop().result()
  File "C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py", line 428, in result
    return self.__get_result()
  File "C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I have tried putting this in a main command, adding wait, adding [var for var in results], list(results) etc. but to no avail.

N.B. I am on python 3.6

Ankit Goel
  • 360
  • 1
  • 5
  • 18
  • 1
    You have to move the `master_list = list(results) into the `if` block. This is due to how Windows spawns new processes instead of forking them. As written, each spawned process tries to access the (non existing) `results` variable. I wrote a longer answer descibing this here: https://stackoverflow.com/a/49345445/7738328 – JohanL Jul 11 '21 at 05:25
  • I added the master_list inside the indent block, but the same error continues - albeit it now occurs on the print (master_data) line outside the block – Ankit Goel Jul 11 '21 at 05:53
  • 1
    You have to keep all the code inside the `if` block to avoid this issue. – JohanL Jul 11 '21 at 06:16
  • Done. It works if all code is in the if block! However, I see several prints of code outside the if block. For eg: if I am printing 'start of process' before the if block (added in question), it gets printed 10 times! Questions: (a) Am I wasting resources / processes (Eg: __mp_main()) that are being executed but failing in the if statement? (b) Am I really achieving multi-processing in this approach? – Ankit Goel Jul 11 '21 at 06:55
  • 1
    Try to put all your code in the ``__name__ == '__main__'`` condition. For example, print is out of this condition. – ThisIsMatin Jul 11 '21 at 06:58
  • Understood! that way I won't be wasting any resources!! It's a long code, so I couldn't relate to putting everything in __name__==__main__ earlier. Thanks! – Ankit Goel Jul 11 '21 at 07:03

0 Answers0