Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
1
vote
0 answers

ipython parallel wokrs over default set up but not over ssh, why?

I have created a set up to connect via ssh to multiple machines. this is my configuration file: c = get_config() c.IPClusterEngines.engine_launcher_class = 'SSHEngineSetLauncher' Clusters =…
evan54
  • 3,585
  • 5
  • 34
  • 61
1
vote
1 answer

Passing IPython parallel cluster object into custom class for batch execution

I am a novice programmer trying to use python for scientific programming. I think these posts (How to work with interactively-defined classes in IPython.parallel? and ipython parallel push custom object) touches on a similar issue but are not useful…
Sang
  • 535
  • 5
  • 14
1
vote
1 answer

Executing some function on IPython is slower than a normal python function

I'm testing some functionalities of ipython and I'm think I'm doing something wrong. I'm testing 3 different ways to execute some math operation. 1st using @parallel.parallel(view=dview, block=True) and function map 2nd using single core function…
xmn
  • 21
  • 3
1
vote
1 answer

AWS Spot Instances and ipcluster plugin

Currently what does the ipcluster plugin do when AWS shuts down one or more of the spot instance nodes? Is there any mechanism to re-start and then re-add these nodes back to the IPython cluster?
Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120
1
vote
1 answer

IPython Parallel LoadBalancedView GIL

I use a loadbalancedview from Ipython.parallel to call a function on an iterable, ie from IPython.parallel import Client from functools import partial rc = Client() lview = rc.load_balanced_view() lview.block = True def func(arg0, arg1, arg2): …
sheridp
  • 1,386
  • 1
  • 11
  • 24
1
vote
1 answer

IPython parallel LoadBalancedView task assignment particulars

I have a few questions about the IPython parallel LoadBalancedView class which I couldn't find in the docs. How does a LoadBalancedView deal with nodes entering and exiting the pool? Does it recalculate loads and reassign? Otherwise once nodes…
lollercoaster
  • 15,969
  • 35
  • 115
  • 173
1
vote
0 answers

Extending IPython task scheduler for custom load balancing

I want to be able to implement my own load-calculating scheme for IPython parallel's LoadBalancedView, which uses a TaskScheduler. I'm piecing through the source, and this looks like a likely…
lollercoaster
  • 15,969
  • 35
  • 115
  • 173
1
vote
0 answers

non DAG task dependencies in LoadBalancedView in iPython

I want to train a lot of models in parallel using ipython parallel with LoadBalancedView. However I want the constraint that after each task is done, that the particular node must "check" with another node (let's call it the task arranger) to make…
lollercoaster
  • 15,969
  • 35
  • 115
  • 173
1
vote
1 answer

"execution_count" error when running a job on a remote IPython cluster

I am running an IPython cluster (SSH) on a remote Linux machine, and I am using Mac OS X with IPython to use that cluster. In IPython on Mac I write: from IPython.parallel import Client c = Client('~/ipcontroller-client.json',…
Sleepyhead
  • 1,009
  • 1
  • 10
  • 27
0
votes
1 answer

How do I get ipyparallel to print status updates while in wait_interactive() in Jupyterlab?

I have the following code blocks, each in JupyterLab (this works as expected in Jupyter, but not Jupyter-Lab): def work(sleepTime): import time import datetime start = datetime.datetime.now() time.sleep(sleepTime) return "Started…
Jared
  • 25,520
  • 24
  • 79
  • 114
0
votes
1 answer

AttributeError: module 'ipyparallel' has no attribute 'Cluster'

I am going through the tutorial to learn ipyparallel and while doing so, I got the error: AttributeError: module 'ipyparallel' has no attribute 'Cluster' I uninstalled and reinstalled the package but the error persisted, does anyone have any tips…
JR222
  • 1
  • 2
0
votes
1 answer

How to use `client.start_ipython_workers()` in dask-distributed?

I am trying to get workers to output some information from their ipython kernel and execute various commands in the ipython session. I tried the examples in the documentation and the ipyparallel example works, but not the second example (with…
SultanOrazbayev
  • 14,900
  • 3
  • 16
  • 46
0
votes
1 answer

Ipython Widgets (how to make a timer)

I made this timer widget that works the way I want, but it locks up the notebook so I can't execute other code cells at the same time. Here's an example of what I mean: Any ideas for getting the widget to execute in the background? In case it…
David Skarbrevik
  • 675
  • 3
  • 9
  • 19
0
votes
1 answer

Parallelizing subgraph tasks in Python

This might be a naive question but I've really tried searching multiple resources: multiprocessing and ipyparallel but these seem to be lack of appropriate information for my task. What I have is a large directed graph G with 9 million edges and 6…
Zhiya
  • 610
  • 2
  • 7
  • 22
0
votes
0 answers

Code gets hung when using multiprocessing

I want to run a simple function across seperate cores on my computer. My computer has four cores. To start with, a simple function: def exp(x): return x**2 now I want to give this function four separate numbers, and use a different core to…
Stefano Potter
  • 3,467
  • 10
  • 45
  • 82
1 2 3
…
12 13