Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
0
votes
0 answers

When I use a sklearn parallel module "from cdnmf_fast import _update_cdnmf_fast", its parallelization is not working

(This is my first time of asking a question, if you think the tags or description have something wrong, please tell me, thank you!) I'm doing a work on Matrix Factorization and use the module sklearn.decomposition.nmf the module uses another…
WooLaw Ren
  • 31
  • 2
0
votes
1 answer

How to import ipyparallel function from user defined package, Python

I have a python script that contains a number of user defined functions that I have set up a as package locally. I can run all of the functions in the input_processing.py script except for the one that I intend to run in parallel. Here is the…
Ben Miller
  • 391
  • 3
  • 5
0
votes
1 answer

How can I set the pythonpath and path of an ipengine (using ipyparallel)?

Using Windows / ipython v6.0.0 I am running ipcontroller and a couple of ipengines on a remote host and all appears to work fine for simple cases. I try to adjust the pythonpath on the remote host (where the ipengines run) such that it can locate…
user1515250
  • 85
  • 1
  • 5
0
votes
1 answer

Start IPython Parallel from within another Python script

Suppose I have two Python files test.py from ipyparallel import Client def hi(a): return b + (a * 2) def run(): b = 3 client = Client() view = client[:] view.push({'b':b}) results = view.map(hi, [0,1,2,3,4]) for r in…
gman9732
  • 77
  • 2
  • 10
0
votes
1 answer

Notification when a new task starts in ipython parallel

A couple of questions related to best practices with ipyparallel. I'm attempting use it to implement a Monte Carlo framework for a model that takes ~15 to run. The idea is to run N engines (via SLURM) and have a "master" process that queues all the…
Rich Plevin
  • 144
  • 1
  • 1
  • 3
0
votes
1 answer

ipyparallel with mpi cannot find engines

I'm trying to get a basic ipyparallel environment working using mpi4py as described in the ipyparallel documentation. After starting the ipcluster, I load ipython and try to create a client but it has no IDs and accessing the directview returns a…
m3wolf
  • 114
  • 4
0
votes
1 answer

Implement parallel processing of for loop

Looking to make the following code parallel- it reads in data in one large 9gb proprietary format and produces 30 individual csv files based on the 30 columns of data. It currently takes 9 minutes per csv written on a 30 minute data set. The…
red79phoenix
  • 91
  • 3
  • 14
0
votes
1 answer

ipyparallel displaying "registration: purging stalled registration"

I am trying to use the ipyparallel library to run an ipcontroller and ipengine on different machines. My setup is as follows: Remote machine: Windows Server 2012 R2 x64, running an ipcontroller, listening on port 5900 and ip=0.0.0.0. Local…
shreyashk
  • 97
  • 7
0
votes
1 answer

Make 2 functions run at the same time and in parallel?

I have an array myArray = array(url1,url2,...,url90) I want to execute this commande 3 times in parallel scrapy crawl mySpider -a links=url and each time with 1 url, scrapy crawl mySpider -a links=url1 scrapy crawl mySpider -a links=url2 scrapy…
parik
  • 2,313
  • 12
  • 39
  • 67
0
votes
1 answer

IPython parallel DirectView.apply does not run in parallel

For some reason I cannot get IPython parallel function DirectView.apply() to really parallelize a function call. DirectView.map() works as expected. Am I doing something wrong here? A working example script import time from datetime import…
truhanen
  • 563
  • 6
  • 9
0
votes
1 answer

Tasks that throw exceptions block IPython-Parallel queue

I am using IPython-Parallel's load_balanced_view and map to perform some simulations. I use the asynchronous version of the view so it doesn't block. Whenever the function applied by mapthrows an exception, the error property of the asynchronous map…
martinzellner
  • 11
  • 2
0
votes
1 answer

subprocess.Popen works outside but not inside ipyparallel?

I'm trying to parallelize some code from here using ipyparallel. In short, I can make functions that work fine outside of apply_sync(), but I can't seem to get them to work inside it (I swear I had this working earlier, but I can't find a version of…
kaz
  • 675
  • 2
  • 5
  • 13
0
votes
1 answer

ipcluster on Sun Grid Engine has only ranks 0

I set up an IPython parallel ipcluster to use the Sun Grid Engine and things seem to work fine: ipcluster start -n 100 --profile=sge 2016-07-15 14:47:09.749 [IPClusterStart] Starting ipcluster with [daemon=False] 2016-07-15 14:47:09.751…
John Smith
  • 1,059
  • 1
  • 13
  • 35
0
votes
1 answer

which core runs the session during concurrent computing

I opened a ipython session, started ipcluster with 4 engines, let engine[0] continuously work in a non-blocking mode. Now I have some computation to do in the ipython session. How do I know and make sure that the computation in this session is not…
Edward
  • 3
  • 2
0
votes
0 answers

IPython parallel on SGE cluster : Scikit-learn SVC parallel import causes engines to crash

I use a SGE cluster with IPcontroller running on the head node, and ~50 engines running on the other nodes (submitted using QSUB). The engines are able to connect and register with the controller without any issues. I can also connect to the head…
KartMan
  • 369
  • 3
  • 19
1 2 3
…
12 13