Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
1
vote
0 answers

Converting Ipyparallel to Dask - Where is the Web UI

I have set up a cluster with ipyparallel on my local machine and wish to convert it to a dask cluster. My primary motivation for this is to be able to use the web ui with dask (via bokeh) to monitor the performance. When I convert the ipyparallel…
Rowan_Gaffney
  • 452
  • 5
  • 17
1
vote
0 answers

Specify custom kernel when using ipcluster

I have a custom Jupyter kernel. It is specified using a kernel.json kernelspec. An extract from kernel.json shows how to start the custom kernel: source /opt/conda/bin/activate gdb_ipykernel && exec /opt/conda/envs/gdb_ipykernel/bin/python -m…
dwjbosman
  • 906
  • 9
  • 33
1
vote
0 answers

debugging ipyparallel applications

Thinking of using ipyparallel to develop a machine learning algorithm on a cluster, mainly using pandas, scikit-learn and numpy. What are the recommended debugging techniques. Is it possible to attach PyCharm to one of the engines to debug the…
Emre Tezel
  • 11
  • 2
1
vote
0 answers

Is ipython or numpy secretly parallelizing matrix multiplication?

So the case is the following: I wanted to compare the runtime for a matrix multiplication with ipython parallel and just running on a single core. Code for normal execution: import numpy as np n = 13 dim_1, dim_2, dim_3, dim_4 = 2**n, 2**n, 2**n,…
SourBitter
  • 157
  • 2
  • 11
1
vote
0 answers

Why can't ipyparallel push/pull when using cloudpickle?

I came across this issue while using ipyparallel. When I try to use cloudpickle, it appears I cannot push or pull globals anymore. Does anybody know the reason or a way around this? In general, I want to be able to send a wrapper around a function…
user1620443
  • 784
  • 3
  • 14
1
vote
1 answer

how are the multiprocessing and threading and thread pooling working

https://code.tutsplus.com/articles/introduction-to-parallel-and-concurrent-programming-in-python--cms-28612 From this link I have studied, I have few questions Q1 : How thread pool (Concurrent) and threading are different here? why do we see the…
NarenSuri
  • 51
  • 4
1
vote
1 answer

passing different arguments for ipython parallel clients

I am learning parallel computation in ipython. I came across an example, from ipyparallel import Client rc = Client() rc.block = True print(rc.ids) def mul(a,b): return a*b dview = rc[:] print(dview.apply(mul, 5, 6)) print(rc[0].apply(mul, 5,…
Kris
  • 9
  • 7
1
vote
0 answers

How do I debug iPython Parallel engine death?

I've got a cluster of engines running. When I set them to work on a long running calculation, after a couple of minutes they just seem to 'die' silently, one by one, until the calculation stalls indefinitely as there are no more engines left to…
cjm2671
  • 18,348
  • 31
  • 102
  • 161
1
vote
1 answer

Ipyparallel error with directview

When running this code, I have this error : from ipyparallel import error, AsyncHubResult, DirectView as dv, Reference @dv.parallel(block=True) def np_std_par(x): return np_std(x) TypeError: unbound method parallel() must be called with…
tensor
  • 3,088
  • 8
  • 37
  • 71
1
vote
0 answers

How to handle KeyError with asynchronous load_balanced_view results in ipyparallel?

Using the customresults.py file in the ipyparallel examples directory, but with my own heavy computation that returns a giant Python dict, I'm…
reckoner
  • 2,861
  • 3
  • 33
  • 43
1
vote
0 answers

load_balanced_view not working properly

I have some code that used to work with ipyparallel using a load_balanced_view. I used to be able to: from ipyparallel import Client rcAll = Client() lbvAll = rcAll.load_balanced_view() for anInpt in allInpt: lbvAll.apply(doAll,…
user3645323
  • 31
  • 1
  • 1
  • 3
1
vote
2 answers

recover values from ipyparallel map if a single evaluation failed

I run a quite time consuming (>3 days) simulation using ipyparallel's map function like that in Jupyter Notebook from ipyparallel import Client rc = Client() lview = rc.load_balanced_view() ar = lview.map(runsimulation, parameter) and I really need…
the_architect
  • 171
  • 9
1
vote
1 answer

Problems while applying a function to each element's content of a directory in python?

I am trying to obtain the content of several .pdf files from a directory in order to transform them to text with tika library, however I believe that I am not reading the .pdf file objects correctly. This is what I tried so far: Input: for filename…
tumbleweed
  • 4,624
  • 12
  • 50
  • 81
1
vote
0 answers

How to configure ipyparallel over IPv6 with CJDNS

I've been dabbling in ipyparallel, and I must give props to their team. Got it running on my local network, but the next step eluded my skills. Can't get ipcontroller and ipengine to converse over a subnet of the Internet built on IPv6 with cjdns.…
Fred Furst
  • 11
  • 2
1
vote
1 answer

Ipyparallel module load on a cluster extremely slow

I have created a 48-node cluster consisting of host0 to host47 (all nodes are g2.2xlarge Amazon EC2 instances with no NFS). According to https://ipyparallel.readthedocs.io/en/latest/process.html, I have created a controller on host0 and 47 engines…
cryo111
  • 4,444
  • 1
  • 15
  • 37
1 2 3
…
12 13