Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
2
votes
1 answer

How to efficiently chain ipyparallel tasks and pass intermediate results to engines?

I am trying to chain several tasks together in iPyParallel, like import ipyparallel client = ipyparallel.Client() view = client.load_balanced_view() def task1(x): ## Do some work. return x * 2 def task2(x): ## Do some work. return x…
nocte107
  • 273
  • 2
  • 10
2
votes
1 answer

How to correctly import modules on engines in Jupyter Notebook for parallel processing?

I wish to run an embarrassingly parallel function that creates plots (and eventually will save them to a file) using Jupyter Notebook with Python (edit - I found a much simpler way to do exactly this here). I'm trying the simplest version possible…
gota
  • 2,338
  • 4
  • 25
  • 45
2
votes
1 answer

ipyparallel view.sync_import does not bind to additional name

i have the following issue with ipyparallel (using python 3.4). The view.sync_imports does neglect the extra module name to which i try to bind them with the 'as' directive. Example: c = Client(profile='slurm') v = c[:] with v.sync_imports(): …
PlagTag
  • 6,107
  • 6
  • 36
  • 48
2
votes
1 answer

IPython-Parallel: Dict literal missing in parallel map

I encountered this problem that any dict literal involved in ipyparallel map is missing and becomes {} in the output result. This problem does not impact dict items added via the update method and dict constructors. To reproduce the problem, you may…
streamliner18
  • 181
  • 2
  • 6
2
votes
0 answers

How to debug and fix iPython parallel's "Kernel Restarting" message on ipython notebook

Every time I run very time consuming task with iPython parallel (ipyparallel), I receive a popup message "Kernel Restarting The kernel appears to have died. It will restart automatically." What is a good way for debugging this situation? I print…
Light Yagmi
  • 5,085
  • 12
  • 43
  • 64
2
votes
0 answers

What is the correct workflow for parallelization: running on a cluster or multiproccesses?

I want to call a function similar to parallelize.map(function, args) that returns a list of results and the user is blind to the actual process. One of the functions I want to parallelize calls subprocess to another unix program that benefits from…
edraizen
  • 23
  • 4
2
votes
0 answers

"Task farming is disabled" or similar socket related problems encountered in IPython.parallel sessions

IPython.parallel's load balanced interface has been great. I used it to run complicated scientific computations on a small cluster environment. Several houndred problems, each taking 5~10 minutes to complete on a PC, were sent to the cluster. I…
dAvid
  • 21
  • 4
2
votes
0 answers

IPython Parallel modify numpy rows in-place

By reading documentation to IPython Parallel it says that the buffers are copied between cores. However, it says that by using track = True one can edit the buffers in place. The following example doesn't work for me, even with track = True: import…
Imanol Luengo
  • 15,366
  • 2
  • 49
  • 67
2
votes
1 answer

how to output results of python parallel computing (ipython-parallel or multiprocessing) to a pandas dataframe?

Simple question: all tutorials I've read show you how to output the result of a parallel computation to a list (or at best a dictionary) using either ipython.parallel or multiprocessing. Could you point me to a simple example of outputing the…
newmathwhodis
  • 3,209
  • 2
  • 24
  • 26
2
votes
0 answers

How can I subclass an ipython cluster controller and engine to add options?

I have the basic PBS Controller and EngineSet Launchers working with my PBS/torque cluster. But now I'd like to pass in additional options through the config file and/or the command line to be rendered in the batch_template file using the {}…
jvd10
  • 1,826
  • 17
  • 17
2
votes
0 answers

Why do I get a module has no attribute __main__ in ipython parallel gather statement?

This is a related question to this one: https://stackoverflow.com/questions/27596463/why-do-i-get-a-cannedarray-object-has-no-attribute-pickled-in-ipython-gather The difference is that now I'm trying to use_dill in ipython parallel and when I try to…
evan54
  • 3,585
  • 5
  • 34
  • 61
2
votes
1 answer

How to do a distributed matrix multiplication in numpy / ipython.parallel?

I saw a tutorial on how to do a distributed calculation: def parallel_dot(dview, A, B): dview.scatter('A', A) dview['B'] = B dview.execute('C = numpy.dot(A, B)') return dview.gather('C') np.allclose(parallel_dot(dview, A, B), …
evan54
  • 3,585
  • 5
  • 34
  • 61
2
votes
1 answer

Import code from local to all remote engines

I am working on parallelizing an sklearn grid search, sweeping three parameters, but I am having trouble refactoring the project to work with ipython.parallel. My current thought approach has been to create a simple function which: Accepts a a…
Cory Dolphin
  • 2,650
  • 1
  • 20
  • 30
2
votes
0 answers

Cannot import numba in ipython cluster engines?

I would like to run a calculation on an IPython cluster that uses a custom numba-fied function to run quickly. I try to synchronize the imports using the sync_imports method of the dview object. The problem is that the engines don't seem to load…
Nathan
  • 508
  • 4
  • 11
2
votes
1 answer

Is it possible to force ipengines to completely reset all local variables and imports?

My workflow is: start ipcontroller/ipengines, then run 'python test_script.py' several times with different parameters. This script includes a map_async call. The ipengines don't recognize changes to the code between calls to the script, and static…
Eric Schmidt
  • 293
  • 3
  • 9
1 2 3
…
12 13