Multiprocessing python within frozen script

Question

I am trying to compile a script utilizing multiprocessing into a Windows executable. At first I ran into the same issue as Why python executable opens new window instance when function by multiprocessing module is called on windows when I compiled it into an executable. Following the accepted answer I adjusted my script such that

from multiprocessing import freeze_support
# my functions
if __name__ == "__main__":
    freeze_support()
    # my script

And this again works perfectly when run as a script. However, when I compile and run it I encounter:

Where I've underlined in green part of the error. This specific line refers to

freeze_support()

in my script. Furthermore, it is not actually encountered on this line, but when my script goes to multiprocess which is something like:

p = multiprocessing.Process(target=my_function, args=[my_list])
p.start()
p1 = multiprocessing.Process(target=my_function, args=[my_list])
p1.start()
p.join()
p1.join()

Is this an error in the multiprocessing module (specifically line 148) or am I misunderstanding the answer I linked, or something else?

I'll also note that the script does work correctly when compiled, but you have to click "OK" on an error message for every multiprocess that is spawned (quite a lot) and every error message is exactly the same. Would this mean i am improperly ending the process with the p.join()?

I've also tried the solution at Python 3.4 multiprocessing does not work with py2exe which recommends adding

multiprocessing.set_executable(os.path.join(sys.exec_prefix, 'pythonw.exe'))

to your script, yet this causes an error in the script form (not even compiled yet) of:

FileNotFoundError: [WinError 2] The system cannot find the file specified

Thanks for the help!

freeze_support documentation: https://docs.python.org/2/library/multiprocessing.html#multiprocessing.freeze_support

score 6 · Accepted Answer · answered Mar 26 '19 at 05:06

This appears to have been a problem for quite some time - I found references going back to 2014, at least. Since it appears to be harmless, the general recommendation is to suppress the error by replacing sys.stdout (and sys.stderr, which is flushed on the next line) with a dummy. Try this:

import os
import sys
from multiprocessing import freeze_support

if __name__ == '__main__':
    if sys.stdout is None:
        sys.stdout = sys.stderr = open(os.devnull, 'w')
    freeze_support()

score 3 · Answer 2 · answered Mar 31 '19 at 19:49

This is not an issue of the multiprocessing library or py2exe per se but a side effect of the way you run the application. The py2exe documentation contains some discussion on this topic:

A program running under Windows can be of two types: a console program or a windows program. A console program is one that runs in the command prompt window (cmd). Console programs interact with users using three standard channels: standard input, standard output and standard error […].

As opposed to a console application, a windows application interacts with the user using a complex event-driven user interface and therefore has no need for the standard channels whose use in such applications usually results in a crash.

Py2exe will work around these issues automatically in some cases, but at least one of your processes has no attached standard output: sys.stdout is None), which means that sys.stdout.flush() is None.flush(), which yields the error you are getting. The documentation linked above has an easy fix that redirects all outputs to files.

import sys
sys.stdout = open(“my_stdout.log”, “w”)
sys.stderr = open(“my_stderr.log”, “w”)

Simply add those lines at the entry point of your processes. There is also a relevant documentation page on the interactions between Py2Exe and subprocesses.

Multiprocessing python within frozen script

2 Answers2