Python, Windows, and Multiprocessing

Question

I have a Python program originally built on and for Linux, which I'm now trying to port over to Windows. I am running the program in a virtual environment which contains all of the dependencies (my program is installed as a wheel with pip install --find-links wheels my_module). The program is launched with

(venv) C:\>venv\Scripts\python.exe -m base_module.Launcher arg1 arg2

The base_module loads my module as interpreted by the arguments provided, and his relevant code is:

from multiprocessing.managers import SyncManager
import OtherCustomClass

class BaseModule(object):
    def __init__(self, arg1, arg2):
        self.manager = SyncManager()
        self.manager.start(ignore_interrupt)

def main(argv=None):
    ret = -1
    try:
        basmod = BaseModule(argv[0], argv[1])
        ret = basmod.run()
    except Exception, err:
        print("error: " + str(err))
        print(traceback.format_exc())
    return ret

if __name__ == "__main__":
    exitCode = main(sys.argv[1:])
    sys.exit(exitCode)

This has worked fine in Linux, but on Windows I get the following exception:

Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Python27\Lib\multiprocessing\forking.py", line 380, in main
      prepare(preparation_data)
    File "C:\Python27\Lib\multiprocessing\forking.py", line 505, in prepare
      '__parents_main__', file, path_name, etc
    File "build/bdist.linux-x86_64/egg/base_module/BaseModule.py", line 2, in <module>
ImportError: No module named OtherCustomClass
exception in main:
Traceback (most recent call last):
    File "build/bdist.linux-x86_64/egg/base_module/BaseModule.py", line 12, in main
    File "build/bdist.linux-x86_64/egg/base_module/BaseModule.py", line 7, in __init__
    File "C:\Python27\Lib\multiprocessing\managers.py", line 528, in start
        self._address = reader.recv()
EOFError

The latter EOFError is caused by the unexpected early termination from the forking in SyncManager, where the true error is being unable to import OtherCustomClass. I have confirmed that OtherCustomClass exists in the base_module's folder within venv/lib/site-packages, and this error isn't happening when I launch the module first as Python would never reach the instructions in main() or init if the script wouldn't compile.

I've done some research, and I know this problem has hit other people (often using third party libraries, who fixed the issue without posting the solution). It seems to trace back to Windows' lack of a fork(), and python's handling of multiprocessing on Windows - see also http://docs.python.org/library/multiprocessing.html#windows. But I'm at a loss as to how to fix this.

This is the latest Python 2.7 branch (2.7.8), running on Windows 7 x64.

Do you have an indentation mistake in your code sample? `main` isn't going to work at all in the way you are using it as an instance method of `BaseModule`. — Silas Ray, Sep 25 '14 at 14:46
Whoops, yes. I trimmed out a ton of the code to just capture what was relevant here, that was an oversight — user2093082, Sep 25 '14 at 14:59
@user2093082 Does this reproduce for you with the minimal example? Does it reproduce if you run it outside of virtualenv? — dano, Sep 25 '14 at 16:18
I installed the packages to the primary python install and reproduced outside of the virtual environment using the system python. I will try to replicate with the example code — user2093082, Sep 25 '14 at 16:23
@user2093082 I was able to reproduce it on Windows, actually. I've got a solution for it, though I'm not sure of the root cause quite yet. — dano, Sep 25 '14 at 16:26

score 2 · Accepted Answer · answered Sep 25 '14 at 16:30

You can work around this by using an absolute import for OtherCustomClass:

from base_module import OtherCustomClass

I'm not exactly sure why, but it seems that when multiprocessing spawns a new process and imports your __main__, it's not able to handle the implicit relative import you're using with OtherCustomClass. If you explicitly import it from base_module, it works fine. My guess is that the spawned child process is not recognized as being part of the base_module package, so the implicit import fails, but that's just a guess.

Note that you shouldn't be using implicit relative imports anyway (they're altogether removed from Python 3), so switching to an absolute import isn't a bad thing.

Also of note, that doing an explicit relative import works on Python 3.4:

from . import OtherCustomClass

But it fails on Python 2.7:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\python27\lib\multiprocessing\forking.py", line 380, in main
    prepare(preparation_data)
  File "C:\python27\lib\multiprocessing\forking.py", line 495, in prepare
    '__parents_main__', file, path_name, etc
  File "C:\Users\oreild1\Desktop\base_module\Launcher.py", line 5, in <module>
    from . import OtherCustomClass
ValueError: Attempted relative import in non-package
error:
Traceback (most recent call last):
  File "C:\Users\oreild1\Desktop\base_module\Launcher.py", line 18, in main
    basmod = BaseModule(argv[0], argv[1])
  File "C:\Users\oreild1\Desktop\base_module\Launcher.py", line 10, in __init__
    self.manager.start()
  File "C:\python27\lib\multiprocessing\managers.py", line 528, in start
    self._address = reader.recv()
EOFError

This is perfect - I had originally imported by filename (which worked in Linux) but as we discovered didn't work in windows, probably due to forking. Importing the module and selecting the class from there works. — user2093082, Sep 30 '14 at 18:10

Python, Windows, and Multiprocessing

1 Answers1

Linked