Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
1
vote
1 answer

IPython Notebook with remote ipyparallel Controller

I'm currently trying to setup a remote cluster on a group of servers I own using the ipyparallel library. I figured that if I share the $IPYTHONDIR between all ipcontrollers, ipengines and notebook that everything would just connect and work, but…
Jeff
  • 873
  • 2
  • 10
  • 16
1
vote
0 answers

start ipcluster with just one batch job submission

Have access to a big compute cluster that uses MPI and SLURM. I successfully got a parallel IPython ipcluster to run using the Slurm job submission system. I can start it for example with ipcluster start -n 320 --profile=slurm And it…
John Smith
  • 1,059
  • 1
  • 13
  • 35
1
vote
1 answer

Ipyparallel slow execution with scatter/gather

Context: I have an array that I have scattered across my engines (4 engines at this time), want to apply a function to each point in the array for an arbitrary number of iterations and gather the resulting array from the engines and perform analysis…
Peter
  • 11
  • 1
1
vote
1 answer

Accessing multiple nodes in a mpi cluster using ipython

This is a continuation of the thread ipython-with-mpi-clustering-using-machinefile. It is slightly more focused and hopefully clearer as to what the issue might be. I have 3-nodes running as a cluster using mpich/mpi4py, a machinefile and all…
horcle_buzz
  • 2,101
  • 3
  • 30
  • 59
1
vote
1 answer

ipyparallel, how to set process numbers of a node

I know I can start my engine by ipengine --profile=/path/to/ipcontroller-engine.json. And there are len(c.ids) processes. But, how can I set the numbers of processes running on every remote node? It seems that ipengine…
GoingMyWay
  • 16,802
  • 32
  • 96
  • 149
1
vote
1 answer

ipyparallel imports: import own module created with cython

I have a problem in synchronising imports between ipyparallel engines. I want to run stochastic simulations in parallel. For this I have written a function named 'gillespie'. I have two versions of the function, one using Python and to gain some…
baumgast
  • 11
  • 2
1
vote
1 answer

Monitoring status of ipyparallel completion

I have a function that takes a while to calculate, and has to be iterated >20k times with two varying parameters: from ipyparallel import Client import numpy as np m_array = np.arange(0, 101, 1) s_array = np.arange(0, 201, 1) rc =…
DataSwede
  • 5,251
  • 10
  • 40
  • 66
1
vote
1 answer

ipython parallel and own class

I'm working on a class that handles numeric operations to perform with a data array. Unfortunately I do not get to work by applying a function of the class to the created direct view. I get the error: , copy) 163 assert len(bufs) >= 2, "not…
Markus
  • 21
  • 3
1
vote
1 answer

IPython.parallel.Client() not working

There is no problem when I import IPython.parallel -->import IPython.parallel However, when I try this: rc=IPython.parallel.Client() I get the following error: /home/mycomputer/local/lib/python2.7/site-packages/IPython/utils/shimmodule.pyc in…
CuriousCoder
  • 491
  • 5
  • 9
1
vote
1 answer

Stopping ipcluster engines (IPython Parallel)

How do I stop the engines that have been started with: ipcluster engines --n=8 --daemonize The currently running processes look like: /usr/bin/python /usr/local/bin/ipcluster engines --n=8 --daemonize /usr/bin/python -m IPython.parallel.engine…
Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120
1
vote
1 answer

ipython parallel cluster parallel decorator and higher order functions

I would like to take an existing function (from scikit-learn for example: specifically the "predict" function), and apply it using multiple cores to some dataset. My first naive approach: def parallel_predict(classifier): …
Andrew Spott
  • 3,457
  • 8
  • 33
  • 59
1
vote
1 answer

Distributed computing instance usage with starcluster Ipython parallel plugin

I am using starcluster with Ipython plugin . When I run a Kmeans clustering from Ipython notebook with load balance mode. Its always the Master with 100% CPU usage constantly. And the other EC2 instances never takes the load. I tried with large…
Hanuma Tej
  • 17
  • 4
1
vote
1 answer

How to import modules in IPython Clusters

I am trying to import some of my personal modules into my IPython Clusters. I am using Anacondas on Windows Vista 64 bit from IPython.parallel import Client rc = Client() dview = rc[:] with dview.sync_imports(): import lib.rf It is…
Ginger
  • 8,320
  • 12
  • 56
  • 99
1
vote
1 answer

ipython parallel and non-copying sends of numpy arrays

I'm trying to understand what happens with IPython parallel's non-copying send/receive of numpy arrays. I understand that the non-copying receive of a message is read-only, but this leads me to expect that the numpy array that I receive acts like a…
AJ Friend
  • 703
  • 1
  • 7
  • 16
1
vote
1 answer

Run bash commands in parallel using iPython

I'm using iPython notebook on a 16 core machine (so there are 15 "engines"). If I run this cell ... %%px %%bash echo 'hi' > file1.txt ... then the result is one file called file1.txt that is written to disk 15 times. What I really want is to…
retsreg
  • 13
  • 3
1 2 3
…
12 13