Best way to pass local variables to ipyparallel cluster

Question

I'm running a simulation in an ipython notebook that is composed of seven functions that are dependent of each other, and requires 13 different parameters. Some of the functions are called within other functions to allow one function to run the entire simulation. The simulation involves manipulating two parameters for a total of >20k iterations. Two simulations can be run asynchronously. Since each iteration is taking ~1.5 seconds, I'm investigating parallel processing.

When I first tried ipyparallel, I got a global name not defined error. Makes sense that local objects can't been found a worker. In an effort to avoid spending quite a bit of time going down a rabbit hole, what would be the easiest way to pass a whole bunch of objects to all of the workers? Are there other gotchas to consider when using ipyparallel in this way?

score 8 · Accepted Answer · edited May 23 '17 at 12:10

There is a bit more detail in this related question, but the gist is: interactively defined modules resolve in the interactive namespace (__main__), which is different on the engine and client. You can send functions to the engine with view.push(dict(func=func, func2=func2)), in which case they will be found. The alternative is to define your functions in a module or package that you ensure is installed on all the engines.

For instance, in a script:

def bar(x):
    return x * x

def foo(y):
    return bar(y)

view.apply(foo, 5) # NameError on bar
view.push(dict(bar=bar)) # send bar
view.apply(foo, 5) # 25

Often when using IPython parallel from a notebook or larger script, one of the early steps is seeding the namespace of the engines:

rc[:].push(dict(
    f1=f1,
    f2=f2,
    const=const,
))

If you have more than a few names to push this way, it might be time to consider defining these functions in a module, and distributing that instead.

I am in a similar situation. What if instead of passing a function, I wanted to push instance of an object? How can I access the object and its methods in the engines? — leonard, Feb 05 '18 at 22:43

Best way to pass local variables to ipyparallel cluster

1 Answers1