Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
4
votes
1 answer

Can a IPython parallel controller both have local and remote ipengines?

The IPython parallel docs mention: c = Client(profile='myprofile') or c = Client('/path/to/my/ipcontroller-client.json') for local ipengines (IIUC) and c = Client('/path/to/my/ipcontroller-client.json', sshserver='me@myhub.example.com') if my…
K.-Michael Aye
  • 5,465
  • 6
  • 44
  • 56
4
votes
0 answers

Dealing with IPython Cluster Node Death

When using the cluster mode in Ipython, how does the master deal with the death of a node machine or process? Specifically if I do an asynchronous apply and some machines die, is there any mechanism to detect this and alert me (by alert I mean is…
Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120
4
votes
2 answers

ipython notebook : how to parallelize external script

I'm trying to use parallel computing from ipython parallel library. But I have little knowledge about it and I find the doc difficult to read from someone who knows nothing about parallel computing. Funnily, all tutorials I found just re-use the…
jrjc
  • 21,103
  • 9
  • 64
  • 78
4
votes
1 answer

Minimizing a function with scipy.optimize parallely with ipython

I found a bunch of questions regarding similar issues (this one, for example), and the problem with pickling a closure seems to be all around when dealing with iPython.Parallel but I couldn't get around this particular issue anywhere. So my problem…
Alex S
  • 1,027
  • 1
  • 10
  • 17
4
votes
1 answer

Is it worth using IPython parallel with scipy's eig?

I'm writing a code which has to compute large numbers of eigenvalue problems (typical matrices dimension is a few hundreds). I was wondering whether it is possible to speed up the process by using IPython.parallel module. As a former MATLAB user and…
MKK_
  • 85
  • 1
  • 6
3
votes
0 answers

How print from ipyparallel engine to the main process?

I implemented a parallel function using the ipyparallel library. I would like to ask you if there is a way to print from the engine to the main process? Here what I'm trying to do: import ipyparallel as ipp rc = ipp.Client() dview = rc[:] v =…
Borreguin Sanchez
  • 31
  • 1
3
votes
1 answer

ipyparallel Parallel for loop that appends to list

I have a loop which does a bunch of CPU intensive calculations and appends the result to a list, per iteration. How can I have that working in parallel. On C# there are concurrent containers, how does it work in ipyparallel? From the ipyparallel…
Abel Dantas
  • 380
  • 2
  • 17
3
votes
0 answers

Use ipython widgets to manipulate plots, that are constantly updated from within an infinite loop

Within an ipython notebook, using Jupyter I try to do a calculation, thats running for an extended period of time in a while loop. I use pyplot to show the current status of my calculation and I would like to be able to communicate with the…
FMey
  • 31
  • 1
3
votes
0 answers

ipyparallel won't let me push an object to an engine

Trying out ipyparallel, and getting stuck on the very early step of pushing code to my engines. Started up my engines with: ipcluster start -n 4 Then try out putting things on the engines with: import ipyparallel as ipp from scipy.interpolate…
I.P. Freeley
  • 885
  • 1
  • 9
  • 19
3
votes
2 answers

ipyparallel Exception: "You have attempted to connect to an IPython Cluster but no Controller could be found"

I've just started using ipyparallel, I'm using VS2017 and importing it as; import ipyparallel as ipp And then attempting to start it using; def main(): rc = ipp.Client() if __name__ == "__main__": main() in the python window I get the…
Robbie Cooper
  • 483
  • 2
  • 5
  • 13
3
votes
0 answers

sync_imports for Ipyparallel with multiple path dependencies

I would like to import a class, lets call it MyClass, and assume it is stored in a file MyClass.py. However, the class itself depends on various files distributed over several folders. So if I want to import that class in a Python session, I would…
user56643
  • 363
  • 1
  • 3
  • 12
3
votes
0 answers

How do I avoid restarting iPython parallel clusters every time I restart the Jupyter kernel?

I am running iPython Parallel. Every time I restart the main notebook kernel, I have to restart the cluster as well. Without doing so, I get a NoEnginesRegistered error (in other words, the directview cannot see the engines). This is quite tedious…
cjm2671
  • 18,348
  • 31
  • 102
  • 161
3
votes
0 answers

In an IPython cluster how can I gracefully interrupt a worker

I want to run some jobs in a cluster, but I want to be able to kill the job if it is taking too long. Can I do this gracefully from the client, and still have the worker available to do more jobs? My scenario is that I want to investigate how…
Tony
  • 339
  • 3
  • 10
3
votes
1 answer

How to stop an IPython cluster without the ipcluster command

I don't start my IPython cluster with the ipcluster command but with the individual commands ipcontroller and ipengine because I use several machines over a network. When starting the cluster with the ipcluster command, stopping the cluster is…
PierreE
  • 675
  • 1
  • 11
  • 23
3
votes
0 answers

MemoryError sending data to ipyparallel engines

We love Ipython.parallel (now ipyparallel). There is something that bugs me, though. When sending a ~1.5GB pandas dataframe to a bunch of workers, we get a MemoryError if the cluster has many nodes. It looks like there are as many copies of the…
Pierre D
  • 24,012
  • 7
  • 60
  • 96
1 2
3
…
12 13