I have three files in a folder:
MultiProcFunctions.py
The idea is to take any function and parallelize it
import multiprocessing
from multiprocessing import Manager
def MultiProcDecorator(f,*args):
"""
Takes a function f, and formats it so that results are saved to a shared dict
"""
def g(procnum,return_dict,*args):
result = f(*args)
return_dict[procnum] = result
g.__module__ = "__main__"
return g
def MultiProcFunction(f,n_procs,*args):
"""
Takes a function f, and runs it in n_procs with given args
"""
manager = Manager()
return_dict = manager.dict()
jobs = []
for i in range(n_procs):
p = multiprocessing.Process( target = f, args = (i,return_dict) + args )
jobs.append(p)
p.start()
for proc in jobs:
proc.join()
return dict(return_dict)
MultiProcClass.py
A file that defines a class which makes use of the above functions to parallelize the sq function:
from MultiProcFunctions import MultiProcDecorator, MultiProcFunction
def sq(x):
return x**2
g = MultiProcDecorator(sq)
class Square:
def __init__(self):
pass
def f(self,x):
return MultiProcFunction(g,2,x)
MultiProcTest.py
Finally, I have a third file that imports the class above and tries to call the f method:
from MultiProcClass import Square
s = Square()
print s.f(2)
However, this yields an error:
File "C:\Python27\lib\multiprocessing\managers.py", line 528, in start
self._address = reader.recv()
EOFError
I am on Windows 7, and also tried:
from MultiProcClass import Square
if __name__ == "__main__":
s = Square()
print s.f(2)
In this case, I got a different error:
PicklingError: Can't pickle <function g at 0x01F62530>: it's not found as __main__.g
Not sure how to make heads or tails of this. I get neither error on Ubuntu 12.04 LTS, where all of this works flawlessly; so the error definitely has to do with how Windows does things, but I can't put my finger on it. Any insight is highly appreciated!
