Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
0
votes
1 answer

IPython IPCluster different path for different nodes

I have two nodes that I'd like to run IPython.parallel engines on. One of them is a linux box, the other runs MacOSX. The directory names are slightly different on the different machines and I can't find a way to start SSHEngines on both machines at…
hanno
  • 6,401
  • 8
  • 48
  • 80
0
votes
0 answers

Controlling prioritization in IPython Parallel

Can one specify any sort of prioritization to tasks submitted to an IPython Cluster? The only hack I know of to accomplish something like this is to carefully maintain a DAG of task dependencies and then, whenever submitting a new prioritized tasks,…
Yang
  • 16,037
  • 15
  • 100
  • 142
0
votes
1 answer

IPython.parallel client is hanging while waiting for result of map_async

I am running 7 worker processes on a single machine with 4 cores. I may have made a poor choice with this loop while waiting for the result of map_async: while not result.ready(): time.sleep(10) for out in result.stdout: print…
Aaron
  • 11
  • 2
0
votes
1 answer

python multiprocessing to a function with multiple functions

My function resembles with the determination of pi value using Monte Carlo integral method. The function basically insert a molecule in random position and estimate the energy. I convert my for loop in a function to operate over multiple cores using…
Muhammad Sajid Iqbal
  • 93
  • 1
  • 9
0
votes
3 answers

multiprocessing to a python function

How I can implement the multiprocessing to my function.I tried like this but did not work. def steric_clashes_parallel(system): rna_st = system[MolWithResID("G")].molecule() for i in system.molNums(): peg_st = system[i].molecule() …
Muhammad Sajid Iqbal
  • 93
  • 1
  • 9
0
votes
1 answer

IPython ipengineapp creation with keyword arguments

I am trying to write a script that will start an new engine. Using some code from IPython source I have: [engines.py] def make_engine(): from IPython.parallel.apps import ipengineapp as app app.launch_new_instance() if __name__ ==…
Lynx-Lab
  • 795
  • 8
  • 17
0
votes
1 answer

Simultaneously using multiple views in the iPython Notebook

I've got a question I'm hoping someone can help me figure out. I'm trying to construct two different parallel views in an iPython notebook. The first view has the processor with ID 0, and the second has all the rest of the processors. I associate a…
Daniel Fenn
  • 3
  • 1
0
votes
1 answer

What is the difference between blocking and non-blocking mode in IPython.parallel in load_balanced_mode()?

In all the examples I found for ipython: it seems that when using a load-balanced view block is set to False. I was wondering why? Also, is it possible to set it to True and what implication would that have? I know this may be a basic question but I…
evan54
  • 3,585
  • 5
  • 34
  • 61
0
votes
1 answer

How to use IPython.parallel for functions with multiple inputs?

This is my first attempt at using IPython.parallel so please bear with me. I read this question Parfor for Python and am having trouble implementing a simple example as follows: import gmpy2 as gm import numpy as np from IPython.parallel import…
evan54
  • 3,585
  • 5
  • 34
  • 61
0
votes
1 answer

IPython parallel and map performance

I have used parallel computing before through MPI (and Fortran :)). I would like to use now the parallel capabilities of IPython. My question is related to the poor performance of the following code, inspired by…
Flavien Lambert
  • 690
  • 1
  • 9
  • 22
0
votes
1 answer

Tell IPython Parallel to use Pickle again after Dill has been activated

I'm developing a distributed application using IPython parallel. There are several tasks which are carried out one after another on the IPython cluster engines. One of these tasks inevitably makes use of closures. Hence, I have to tell IPython to…
t3c
  • 3
  • 4
0
votes
1 answer

parallelization issue using python view library

so i Worte some code with basic structure like this: from numpy import * from dataloader import loadfile from IPython.parallel import Client from clustering import * data = loadfile(0) N_CLASSES = 10 rowmax = nanmax(data.values, 0) rowmin =…
user3720918
  • 57
  • 1
  • 4
0
votes
1 answer

ipengine does not keep running after closing SSH session

I'm trying to run a couple of remote servers with ipcontroller and ipengine. So I start a single process with ipcontroller on "master" node, which I will be running ipython notebook on at the end. Running ipengine on remote hosts goes smoothly…
Arman
  • 927
  • 3
  • 12
  • 32
0
votes
1 answer

Parallel IPython detects closures in list comprehension

I want to parallelize some python code with IPython.parallel and have the problem, that IPython detects closures while executing a parallel function. My environment get's initialized by: from IPython.parallel import Client c = Client() v =…
Stefan
  • 2,460
  • 1
  • 17
  • 33
0
votes
0 answers

IPython Parallel programming dependencies

I am using iPython for some relatively heavy numerical tasks, subsets of which are more or less embarrassingly parallel. The tasks have very simple dependencies, but I'm struggling to work out the best way to implement them. The basic problem is…
Andrew Jaffe
  • 26,554
  • 4
  • 50
  • 59
1 2 3
…
12
13