Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
3
votes
1 answer

Fail to import IPython parallel in Jupyter

I recently made an update of IPython to 4.0.0 and installed Jupyter 4.0.6. I wanted to use Ipython parallel, and after starting the engines in the notebook, I imported: from IPython import parallel And it…
jrjc
  • 21,103
  • 9
  • 64
  • 78
3
votes
1 answer

Starting an IPython notebook MPI cluster

I am having a very hard time trying to start an IPython cluster that has MPI capabilities. First of all i have MPI and MPI4py installed and they are working. I finally figured out how to run an mpi python code from within ipython. However, I am…
bluerubez
  • 300
  • 2
  • 14
3
votes
2 answers

IPython parallel computing vs pyzmq for cluster computing

I am currently working on some simulation code written in C, which runs on different remote machines. While the C part is finished I want to simplify my work by extending it with a python simulation api and some kind of a job-queue system, which…
jrsm
  • 1,595
  • 2
  • 18
  • 39
3
votes
1 answer

ipcluster - can't start more than about 110 ipengines - or maybe some of them die

I'm having difficulty getting ipcluster to start all of the ipengines that I ask for. It appears to be some sort of timeout issue. I'm using IPython 2.0 on a linux cluster with 192 processors. I run a local ipcontroller, and start ipengines on my…
DailRowe
  • 31
  • 2
3
votes
1 answer

IPython Cell Magic to Run this Cell in Client and in All Engines

In IPython for parallel execution we have multiple engines. The jobs are dispatched to them using: %%px cell magics to set up the environment in the clients and then using DirectView map_sync to run the various experiments. The experiments return a…
Frames Catherine White
  • 27,368
  • 21
  • 87
  • 137
3
votes
0 answers

IOError error message in IPython notebook when using StarCluster

I am running a small StarCluster and using it to run an IPython Notebook. Every time I have an error in the code I'm writing in the notebook, I get the following error message added onto the end of the notebook's output: Traceback (most recent call…
Maximilian
  • 7,512
  • 3
  • 50
  • 63
3
votes
1 answer

IPython.parallel ValueError: cannot create an OBJECT array from memory buffer

I'm trying to write a function to be executed in several IPython engines. The function takes a pandas Series as an argument. Each element of the Series is a string, and the whole Series constitutes a corpus for TF.IDF computation. After reading…
herrfz
  • 4,814
  • 4
  • 26
  • 37
3
votes
2 answers

Embarassingly parallel tasks with IPython Parallel (or other package) depending on unpickable objects

I often hit problems where I wanna do a simple stuff over a set of many, many objects quickly. My natural choice is to use IPython Parallel for its simplicity, but often I have to deal with unpickable objects. After trying for a few hours I…
Rafael S. Calsaverini
  • 13,582
  • 19
  • 75
  • 132
2
votes
1 answer

ipyparallel: what is the most reliable way to clear/reset engine namespaces without restarting the cluster?

As far as I understand, an ipython cluster manages a set of persistent namespaces (one per engine). As a result, if a module that is imported by an engine engine_i is modified, killing the main interpreter is not sufficient for that change to be…
Ash
  • 4,611
  • 6
  • 27
  • 41
2
votes
1 answer

Can't run IPython-Parallel in Jupyter Notebook

I'm trying to follow these instructions on using ipyparallel in order to speed up some Python scripts in Jupyter Notebook. When doing import ipyparallel as ipp IPPC = ipp.Client() I get the following error: Using matplotlib backend: TkAgg Waiting…
sarah.connr
  • 21
  • 2
2
votes
2 answers

IPython-Parallel: ipcluster and ipcontroller-client.json not found

I'm trying to follow these instructions on using ipyparallel in order to speed up some Python scripts in Jupyter Notebook. When doing import ipyparallel as ipp IPPC = ipp.Client() I get the following error: Using matplotlib backend: TkAgg Waiting…
ajewzaliefff
  • 21
  • 1
2
votes
0 answers

ipyparallel's LoadBalancedView bloats memory, how can I avoid that?

This issue may be related to https://github.com/ipython/ipyparallel/issues/207 which is also not marked as solved, yet. I also opened this issue here https://github.com/ipython/ipyparallel/issues/286 I want to execute multiple tasks in parallel…
LFish
  • 299
  • 1
  • 6
  • 16
2
votes
0 answers

IPython Cluster using SSH

Having trouble starting cluster on remote machines. When starting cluster “manually” and copy json config files everything works. I can connect with notebook and everything works as expected. But I would like to be able to start cluster over SSH.…
mar5
  • 51
  • 4
2
votes
0 answers

Python parallel programming “No traceback available"

I am trying to run parallel code in python utilizing ipyparallel package. import ipyparallel as ipp # create client & view rc = ipp.Client() dv = rc[:] v = rc.load_balanced_view() def my(i): import numpy as np …
Kernel
  • 591
  • 12
  • 23
2
votes
3 answers

"ipcluster nbextension enable" doesn't have subcommand specified

I have ipyparallel 5.3.0. Its official doc, https://github.com/ipython/ipyparallel#install, instructs the following: To enable the IPython Clusters tab in Jupyter Notebook: ipcluster nbextension enable But I got the following issue hence it…
etlolap
  • 531
  • 1
  • 6
  • 14
1 2 3
…
12 13