Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
0
votes
1 answer

Debug iPython Parallel engines in cluster

Related to this thread... I am trying to track down a bug in which the results from processing on an iPython cluster do not match what happens when the same process is run locally. Even when the iPython cluster is entirely local, and the CPU is…
P-Rod
  • 471
  • 1
  • 5
  • 18
0
votes
2 answers

Errors using Sympy's `lambdify` with `IPython.parallel`

For a numerical computation I am working on, I need to define a large Liouvillian matrix. Rather than program it in element by element, which is tedious and error-prone, I have used Sympy to construct it algebraically, and then use lambdify to make…
Tom Baldwin
  • 978
  • 7
  • 9
0
votes
1 answer

IPython engines returning different results

Hopefully someone can enlighten me without me having to post a lot of confusing code. I am using IPython.parallel to process neural networks. In an attempt to find a bug I decided to send the same network out to each client with the same input…
P-Rod
  • 471
  • 1
  • 5
  • 18
0
votes
1 answer

retrieving python variables from Cell Magic in Ipython notebook from background subprocess

I'm trying to do a quick and dirty querying of my mongodb database using ipython notebook. I have several cells each with its own query. Since mongodb can support several connections I would like to run each query in parallel. I thought an ideal way…
jwillis0720
  • 4,329
  • 8
  • 41
  • 74
0
votes
1 answer

unable to push dataframe into ipython parallel engine

I pushed a pandas dataframe object of 1MM rows and 35 columns into directView of ipython parallel engine. However, I am having trouble pushing this data (or even an empty dataframe) into the engine as my function fails to print the length of the…
cs_newbie
  • 1,959
  • 1
  • 15
  • 16
0
votes
1 answer

Retrieve engine id in function

I am using ipython parallel to schedule a large number of job using load_balanced_view. Each job uses subprocess.Popen to run a code and retrieve stdout and stderr. Then I want to save them into a log file. This is an example of the structure of the…
Francesco Montesano
  • 8,485
  • 2
  • 40
  • 64
0
votes
1 answer

Asynchronous evaluation in ipython parallel

Since the awesome 1.0.0 release I've been playing around with iPython parallel interface. What I'm trying to do is to set up a asynchronous stochastic gradient descent system. The way I see it, I want to send a function to all the nodes and get the…
Alex S
  • 1,027
  • 1
  • 10
  • 17
0
votes
1 answer

Retrieving results from IPython parallel hub database

I've been running parallel jobs on a SGE cluster using IPython parallel. I submit my jobs and retrieve the results from the hub database (SQlite) at a later time when all the jobs have finished, using the jobs message ID. This worked fine till my…
KartMan
  • 369
  • 3
  • 19
0
votes
2 answers

speedup postgresql to add data from text file using Django python script

I am working with server who's configurations are as: RAM - 56GB Processor - 2.6 GHz x 16 cores How to do parallel processing using shell? How to utilize all the cores of processor? I have to load data from text file which contains millions of…
user2423706
  • 159
  • 4
  • 15
0
votes
1 answer

ipython 0.13 zmq errors

I encounter weird behavior of an ipython cluster. The calculations finish, but many results never reach the client (and the engines just idle after finishing their first calculation). I suspect something is wrong with zmq because 1) from time to…
Anton Akhmerov
  • 154
  • 9
-1
votes
1 answer

Again Real-time output from engines in IPython parallel?

I have the same question as in this stackoverflow post and my ipython (0.13.1) does seem to have the 'Result not ready' error, so that I used the following change : def wait_watching_stdout(ar, rc, dt=1, truncate=1000): while not ar.ready(): …
Johann cohen-tanugi
  • 127
  • 10
1 2 3
…
12
13