Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
6
votes
1 answer

Change IPython 3 for Python 3 kernel to python2 for the cluster too

I have IPython 3 installed for Python 3 in order to work with Jupyterhub. Now I'm able to use notebooks with a Python2 kernel, because I created /usr/local/share/jupyter/kernels/python2/kernel.json with: { "argv": ["python2", "-m",…
johnbaltis
  • 1,413
  • 4
  • 14
  • 26
6
votes
0 answers

logging with IPython parallel

I am trying to setup logging when using IPython parallel. Specifically, I would like to redirect log messages from the engines to the client. So, rather than each of the engines logging individually to their own log files, as in IPython.parallel -…
Jan Kwakkel
  • 78
  • 6
5
votes
0 answers

How to parallelize with jupyter and sklearn?

I'm trying to parallelize the GridSearchCV of scikit-learn. It's running on a jupyter (hub) notebook environment. After some research I found this code: from sklearn.externals.joblib import Parallel, parallel_backend, register_parallel_backend from…
ScientiaEtVeritas
  • 5,158
  • 4
  • 41
  • 59
5
votes
1 answer

Is there any reason to use Ipyparallel for common python script (not ipython notebook) over multiprocessing module?

Is there any reason to use Ipyparallel for common python script (not ipython notebook)?
Philipp_Kats
  • 3,872
  • 3
  • 27
  • 44
5
votes
1 answer

Using IPython Parallel on the Sun Grid Engine

I'm trying to use IPython Parallel for a very common scenario, where I want to run simulations on a cluster running Sun Grid Engine, and I can't find a reliable way to do this. Here's what I am trying to do: I want to run numerical simulations…
KartMan
  • 369
  • 3
  • 19
5
votes
0 answers

How can I use %autoreload on ipython parallel engines?

Is it possible to get the magic command %autoreload to work on remote ipengines? The following works: dv.execute('a=blah.test()') dv.get('a') 5 I then change blah to return…
Eric Schmidt
  • 293
  • 3
  • 9
4
votes
1 answer

ipykernel_launcher processes are consuming memory, Not able to kill

What are these zombie ipykernel_launcher process in my machine, which are hogging to much memory: This is output of htop command, but I ps for those processes,(to kill them) I do not see them as: ps -ef|grep ipykernel Not sure, how to get rid of…
sandeepsign
  • 539
  • 6
  • 11
4
votes
2 answers

Starting ipcluster from code

I want to dynamically start clusters from my Jupyter notebook for specific functions. While I can start the cluster and get the engines running, I am having two issues: (1) I am unable to run the ipcluster command in the background. When I run the…
RRC
  • 1,342
  • 2
  • 11
  • 17
4
votes
0 answers

Iterate through AsyncMapResult where *some* tasks have raised an exception

When using an ipyparallel cluster to process tasks in parallel, how do I iterate over the AsyncMapResult when some of the tasks have raised an exception? All I get is the exception, but I can't figure out how to get at results of the tasks that have…
Kal
  • 1,707
  • 15
  • 29
4
votes
2 answers

Can I restart an iPython cluster from a notebook?

I'm just wondering if there is some python code or magics I can execute that will restart the ipython cluster. It seems like every time I change my code, it needs to be restarted.
cjm2671
  • 18,348
  • 31
  • 102
  • 161
4
votes
0 answers

Ipyparallel ssh cluster creates/connects, but does not return workers

I'm trying to setup the ipyparallel cluster over ssh with the following settings specified in the ~/.ipython/profile_ssh/ipcluster_config.py: c.IPClusterEngines.engine_launcher_class = 'SSH' c.IPClusterStart.controller_launcher_class =…
T. Ostasevicius
  • 95
  • 3
4
votes
1 answer

ipyparallel Hub Connection Timeout with parallel CV in scikit learn

I am trying to train multiple RandomForest classifiers using ipyparallel. My design is nested CV loops outer CV with for remove variance, inner GridSearchCV with built in (n_jobs = -1) to find the best estimators (# forests, #trees in Forest) for…
ssm
  • 620
  • 6
  • 24
4
votes
1 answer

itertools.ifilter with IPython Parallel

For some problem [proven to be NP hard] I have no other option but exhaustive search. I have a set of data — for simplicity, S = ['A', 'B', 'C', ... ,'Z'] and want to apply a function f to all subsets of length N < len(S) of this set. I cannot use…
Yury Kirienko
  • 1,810
  • 1
  • 22
  • 32
4
votes
0 answers

Write pandas dataframes to csv using ipython parallel

I am trying to write multiple pandas data frames to a csv using the ipython parallel module as doing so serially is very slow. Here is a small example of what I am trying to do: from IPython.parallel import Client import pandas as pd import numpy…
johnchase
  • 13,155
  • 6
  • 38
  • 64
4
votes
1 answer

Best way to pass local variables to ipyparallel cluster

I'm running a simulation in an ipython notebook that is composed of seven functions that are dependent of each other, and requires 13 different parameters. Some of the functions are called within other functions to allow one function to run the…
DataSwede
  • 5,251
  • 10
  • 40
  • 66
1
2
3
…
12 13