1

I use a loadbalancedview from Ipython.parallel to call a function on an iterable, ie

from IPython.parallel import Client
from functools import partial

rc = Client()
lview = rc.load_balanced_view()
lview.block = True

def func(arg0, arg1, arg2):
    return func2(arg0) + arg1 + arg2

def func2(arg0):
    return 2*arg0

answer = lview.map(partial(func, arg1=A, arg2=B), iterable_data)

Does the fact that func calls func2 make func not be executed in parallel (ie. does the GIL come into play?) I assume that when you call map, each cluster node gets a copy of func, but do they also get copies of func2. Further, does the fact that I use functools.partial cause any problems?

sheridp
  • 1,386
  • 1
  • 11
  • 24

1 Answers1

4

Does the fact that func calls func2 make func not be executed in parallel (ie. does the GIL come into play?)

Not at all. The GIL is not at all relevant here, nor is it ever relevant in the parallelism in IPython.parallel. The GIL only comes up when coordinating threads within each engine, or within the Client process itself.

I assume that when you call map, each cluster node gets a copy of func, but do they also get copies of func2.

It should, but this is actually where your code will have a problem. IPython does not automatically track closures, and code dependencies in the interactive namespace, so you will see:

AttributeError: 'DummyMod' object has no attribute 'func'

This is because partial(func, arg1=A, arg2=B) contains a reference to __main__.func, not the code of the local func itself. When the partial arrives on the engine, it is deserialized, and __main__.func is requested, but undefined (__main__ is the interactive namespace on the engine). You can address this simply by ensuring that func and func2 are defined on the engines:

rc[:].push(dict(func=func, func2=func2))

At which point, your map should behave as expected.

If you instruct IPython to use the enhanced pickling library dill it will get closer to not having to manually send references, but it doesn't cover every case.

minrk
  • 37,545
  • 9
  • 92
  • 87
  • Hi minrk, thanks for the response. What I've found so far is that even without dill, I don't get any errors and the code will run, but it is approximately 10x slower than a non-parallelized version (actual func uses numpy arrays, so I suspect it is passing these back and forth a lot). I'm a bit confused as to how ipython treats functions declared in a module that uses a remote client. I never pushed func2 to the remote clients, so why does that not throw a NameError when it tries to execute it? – sheridp Apr 07 '14 at 13:59
  • Ah I noticed, you specified "...code dependencies in the **interactive** namespace ...". I should mention, this code is actually placed in a module and imported. Does IPython track code dependencies in imported modules? I just did a quick check, and you are right that I get an AttributeError when testing in an interactive session. – sheridp Apr 07 '14 at 14:10
  • It doesn't need to track any dependencies in the module case. pickle stores references when serializing. These deserialize just fine across network boundaries for pretty much everything other than interactively defined names. – minrk Apr 07 '14 at 16:16
  • This is all unrelated to performance. Can you update the question to perhaps a sample that is representative of what you are doing, and mentions the actual goal you are trying to achieve? The functions and their closures probably have nothing to do with that, it is likely just moving the numpy arrays back and forth. – minrk Apr 07 '14 at 16:19
  • I still find it a bit odd that, were I to import the above module in an interactive session, the remote clients have access to func2. Since the RCs are declared _in the module_ they don't import it, nor is func2's code pushed to them (at least not explicitly). It seems strange that, just because I import a module _locally_, all code within the module gets pushed to remote clients declared in the same module. – sheridp Apr 08 '14 at 17:24
  • I agree that this is unrelated to performance. I think the performance hit is just a simple case of parallelism overhead not being worth it if the task itself (func) is not long-running enough. – sheridp Apr 08 '14 at 17:26
  • When you pickle a module function, it does not pickle its code, it only pickles a reference. See the output of `pickle.dumps(module.func)` for instance. Thus, when func is unpickled on the client, that implies an import of func's module, which contains func2. If func's module were not available on the engine, func wouldn't work. – minrk Apr 09 '14 at 18:10
  • Ah I see, thanks for the info. It happens that my local client is running on the same machine as the cluster clients at the moment, and so the import was working without error. I didn't realize that it would automatically import the module at unpickling. Thanks again for clearing this up. – sheridp Apr 09 '14 at 19:35