0

I've got some simulation code I've written using the simPy framework. To run multiple simulations with different parameters I'm unable to use the standard Python multiprocessing module, as it won't successfully pickle all the arguments I need to pass into a simulation instance. I got around this by switching to the pathos multiprocessing module (yay!) -- but the pypy interpreter crashes out when the pathos/starmap combination is invoked.

The pathos/multiprocessing pool invocation is:

with pathos.helpers.mp.Pool() as pool:
   results = pool.starmap(runSim, argsToRun)

argsToRun is a list of parameters specific to that simulation run. Most are boring integers or strings, but one argument is an instance of a simpy Environment, which standard python fails to pickle - this is originally what drove me to move to pathos.

At this point I'm not sure whether this is expected to work (meaning whether anyone would be motivated to even look at it) and if I clear that bridge I'm not sure whether to work with the pathos side, or the pypy side. Running these simulations in Pypy instead of cPython gives me somewhere between 4x and 8x better performance, but I also have a 32-core machine so being able to run multiprocessing is even more important. I'm greedy like that, so I really wanna get both working together and it doesn't seem terribly insane that there would be others who would benefit. ;)

Traceback (most recent call last):
  File "/users/lwobker/scripts/pypy/site-packages/multiprocess/process.py", line 258, in _bootstrap
    self.run()
  File "/users/lwobker/scripts/pypy/site-packages/multiprocess/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/users/lwobker/scripts/pypy/site-packages/multiprocess/pool.py", line 108, in worker
    task = get()
  File "/users/lwobker/scripts/pypy/site-packages/multiprocess/queues.py", line 340, in get
    return _ForkingPickler.loads(res)
  File "/users/lwobker/scripts/pypy/site-packages/dill/_dill.py", line 275, in loads
    return load(file, ignore, **kwds)
  File "/users/lwobker/scripts/pypy/site-packages/dill/_dill.py", line 270, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/users/lwobker/scripts/pypy/site-packages/dill/_dill.py", line 472, in load
    obj = StockUnpickler.load(self)
  File "/users/lwobker/pypy/pypy3.6-7.2.0-linux_x86_64-portable/lib-python/3/pickle.py", line 1070, in load
    dispatch[key[0]](self)
  File "/users/lwobker/pypy/pypy3.6-7.2.0-linux_x86_64-portable/lib-python/3/pickle.py", line 1418, in load_reduce
    stack[-1] = func(*args)
AttributeError: 'Environment' object has no attribute 'Process'
ljwobker
  • 832
  • 2
  • 10
  • 20

1 Answers1

0

I see both /users/lwobker/scripts/pypy and /users/lwobker/pypy/pypy3.6-7.2.0-linux_x86_64-portable. Are they using the same version of whatever library has Environment ?

mattip
  • 2,360
  • 1
  • 13
  • 13
  • In the most humble voice I can muster: "I think so, but I'm not 100% sure. How would I confirm this?" -- I'm a lowly network engineer, not an actual programmer. ;-) – ljwobker Dec 18 '19 at 00:37
  • 1
    `pypy -c "import sys; print(sys.version)"` for both of the executables will tell you what version of pypy you are using. `pypy -c"import simpy; print(simpy.__version__)"` will tell you what version of simpy you are using (assuming `simpy` is a well-behaved package, I have no idea about simpy). A hint: perhaps you want to be pickling some attribute/s of the `Environment` and not the whole object, which might make the pickle smaller and more portable. – mattip Dec 19 '19 at 06:59