Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
0
votes
1 answer

IPython parallel : how to recover job IDs from IPcontroller

I have a server running IP controller and 12 IPengines. I connect to the controller from my laptop using SSH. I submitted some jobs to the controller using the load-balanced view interface (in non-blocking mode) and stored the message IDs in the…
KartMan
  • 369
  • 3
  • 19
0
votes
1 answer

Sharing queues with ipyparallel cluster

I am trying to integrate ipyparallel as an alternative to the multiprocessing in my master - slaves architecture. Namely, currently all processes have two queues: One for tasks from master to slaves One for results from slaves to master. At the…
T. Ostasevicius
  • 95
  • 3
0
votes
1 answer

iPython parallel pushing every objects is messy

I love iPython notebook but on parallel processing with ipyparallel, pushing every local objects into engine with dview.push(dict(...)) seems not effective way. Do you have any effective alternatives to do that? I usually do like: from IPython…
Light Yagmi
  • 5,085
  • 12
  • 43
  • 64
0
votes
1 answer

ipython with MPI clustering using machinefile

I have successfully configured mpi with mpi4py support across three nodes, as per testing of the hellowworld.py script in the mpi4py demo directory: gms@host:~/development/mpi$ mpiexec -f machinefile -n 10 python…
horcle_buzz
  • 2,101
  • 3
  • 30
  • 59
0
votes
1 answer

Split jobs among ipyparallel engines to make external n-threaded calls

The problem Say I have 20 processors available. I want to pass arguments to an external program from IPython that runs best with 4 threads at a time, and use map_async to keep adding jobs until all jobs are finished. Below is example code where I…
Flower-doctor
  • 73
  • 6
0
votes
2 answers

IPython notebook: define a dummy cell magic that simply runs the cell as usual

How can I define a simple cell magic that just executes the cell as if the %%mymagic wasn't there? The context is that we are using the wonderful IPython parallel framework. In some places, we also use its defined %%px magic. But sometimes we'd…
Pierre D
  • 24,012
  • 7
  • 60
  • 96
0
votes
1 answer

IPython ipyparallel map_sync ImportError

I am new to ipyparallel and I want to use this package to implement parallel computing with my machine learning application. The following are a test on ipyparallel, I define a function named add on func.py file, and the main function on test.py…
GoingMyWay
  • 16,802
  • 32
  • 96
  • 149
0
votes
2 answers

IPython parallel does not work for me in IPython 2.2 but did in 2.1

the skeleton code of what I do is from IPython import parallel ..... rcAll = parallel.Client() all_engines = rcAll[:] lbvAll = rcAll.load_balanced_view() .... for anInpt in allInpt: lbvAll.apply(mputil.doAll,…
user3645323
  • 31
  • 1
  • 1
  • 3
0
votes
1 answer

Run IPython Parallel on SGE cluster from local controller (using queue system)

Regarding IPython Parallel, from the Documentation and several posts I found on the Internet, I know I can start a controller on a machine and the engines on another through SSH. However, I'd like to use IPython Parallel on a SGE cluster but…
Rosa
  • 73
  • 4
0
votes
1 answer

How to configure ipython notebook on laptop to point to remote server for execution

Well, my previous attempt to connect Pycharm from laptop to remote server did not see any ray of hope because of tcp/ip issues (which honestly I could not understand much and am still battling with) therefore I am looking to ipython as an…
mnm
  • 1,962
  • 4
  • 19
  • 46
0
votes
0 answers

iPython, running parallel computations, is stuck with results in a strange state

I'm doing some parallel computations, evaluating the goodness of a fit across many regressions. In doing so (running ~60K computations), I've somehow managed to get iPython into a strange state. Pushing objects out to all of the…
clearf
  • 586
  • 4
  • 11
0
votes
1 answer

How to reduce Ipython parallel memory usage

I'm using Ipython parallel in an optimisation algorithm that loops a large number of times. Parallelism is invoked in the loop using the map method of a LoadBalancedView (twice), a DirectView's dictionary interface and an invocation of a %px magic.…
drevicko
  • 14,382
  • 15
  • 75
  • 97
0
votes
1 answer

Are IPython engines independent processes?

From the IPython Architecture Overview documentation we know that ... The IPython engine is a Python instance that takes Python commands over a network connection. Given that it is a Python instance does that imply that these engines are stand…
jxramos
  • 7,356
  • 6
  • 57
  • 105
0
votes
1 answer

Some SSH Engines do not start in Parallel IPython 2.3.0

I am trying to set up a small IPython cluster (This all worked neatly once upon a time with IPython 0.10.0 [sic!]) over ssh in a private network (no security required): 4 nodes alice, bob, carol, dan, each having 4 CPU cores. The controller runs on…
Donludovico
  • 1
  • 2
0
votes
1 answer

Where and how to spin up reusable resources (matlab.engine) for IPython cluster Tasks

I've decided to target the creation of some test bench Python scripts that will exercise various algorithms developed by multiple developers under a variety of environments to all be exercised by IPython's parallel distributed cluster capabilities.…
jxramos
  • 7,356
  • 6
  • 57
  • 105
1 2 3
…
12 13