0

I want to parallelize some python code with IPython.parallel and have the problem, that IPython detects closures while executing a parallel function.

My environment get's initialized by:

from IPython.parallel import Client
c = Client()
v = c.direct_view()

qs = [1,2,3,4]

The function is defined as:

@v.parallel(block=True)
def pbands(qs):
    i = 1 # This should normally be a loop variable inside the function
    valar = [i for j in range(3)]
    return 0

Executing this function as

pbands(qs)

yields the error

ValueError: Sorry, cannot pickle code objects with closures

Replacing the i inside the list comprehension with a literal number or j, doesn't create an error, but this is no suitable solution, because in the real code i will be a loop variable.

Is there a way, to get this list comprehension working?

Stefan
  • 2,460
  • 1
  • 17
  • 33

1 Answers1

2

Python 3 creates closures on list comprehensions. This code wouldn't be a problem on Python 2.

IPython doesn't handle closures. This will be fixed (differently) in IPython 3.0 (adding simple closure support) and 2.1 (fixing the closure check). If you tell IPython to use dill, a lot more things will be serializable, including this particular bit of code:

rc[:].use_dill()

DirectView.use_dill is new in IPython 2.0.

minrk
  • 37,545
  • 9
  • 92
  • 87
  • I tried your solution with `dill 0.2b1`, but this leads to some different errors (`KeyErrors` and `UnpicklingError: pickle exhausted before end of frame`). – Stefan May 05 '14 at 08:04
  • That's an older version of `dill`, maybe try again with version `dill-0.2`. – Mike McKerns May 20 '14 at 17:51