Stack Overflow
Stack Overflow

Questions tagged [parallel-python]

Parallel Python provides mechanism for parallel execution of python code on SMP (systems with multiple processors or cores) and clusters (computers connected via network)

Parallel Python (PP) is a Python module which provides mechanism for parallel execution of python code on SMP (systems with multiple processors or cores) and clusters (computers connected via network).

PP module overcomes the limitations imposed by the GIL (Global Interpreter Lock) on multithreading and provides a simple way to write parallelPpython applications. Internally ppsmp uses processes and IPC (Inter Process Communications) to organize parallel computations.

All the details and complexity of the latter are completely taken care of, and your application just submits jobs and retrieves their results (the easiest way to write parallel applications).

The software written with PP works in parallel even on many computers connected via local network or Internet (clusters). Cross-platform portability and dynamic load-balancing allows PP to parallelize computations efficiently even on heterogeneous and multi-platform clusters.

65 questions
0
votes
0 answers

ParallelPython created in thread gives

Creating a job server gives the error: Python int too large to convert to C long when created in thread (win32), using parallel python version 1.6.5 & Python27. Here is the traceback: Traceback (most recent call last): File…
frage
  • 719
  • 7
  • 15
0
votes
2 answers

how to implement single program multiple data (spmd) in python

i read the multiprocessing doc. in python and found that task can be assigned to different cpu cores. i like to run the following code (as a start) in parallel. from multiprocessing import Process import os def do(a): for i in range(a): …
amg
  • 45
  • 1
  • 8
0
votes
1 answer

Parallel Python: restriction on the number of procceses per core

I'm working with the next code (this is just a part of the full code) in parallel python in a computer with two cores import pp from scheduling import * from numpy import * def sched_pp_process(B,T,D,BL,blocks,number_block,number_core): …
Gerard
  • 1
0
votes
1 answer

way too slow in retrieving results in parallel python (pp)

I used parallel python (pp) package in order to perform a serial parallel processing over my 4-core laptop. Here is a quick summary of the scripts used to do the job. After initializing the parallel python object, I split up my task over 4 jobs and…
user4279562
  • 669
  • 12
  • 25
0
votes
2 answers

Using parallel python with Instance Methods?

Given that code here I try to call : p = ProgressBar(maxval=len(img_paths)) sm = SaliencyMaskSlic() operations = [('img_resize', img_resize), ('sal_mask', sm.transform)] args_list = [{'h_size':258}, {'cropped':True}] pre_pipeline =…
erogol
  • 13,156
  • 33
  • 101
  • 155
0
votes
1 answer

Globals, classes and Parallel Python

I have been reading for a few hours on how globals=globals() works on Parallel Python, and I am still a little bit confused, maybe you can help me... I am writing a code which can basically be summarized as: import pp class Foo(object): def…
MaxG
  • 5
  • 3
0
votes
1 answer

object won't die (still references to it that I can't find)

I'm using parallel-python and start a new job server in a function. after the functions ends it still exists even though I didn't return it out of the function (I used weakref to test this). I guess there's still some references to this object…
Wolfgang Kerzendorf
  • 724
  • 1
  • 9
  • 24
0
votes
1 answer

Getting "ImportError: No module named" with parallel python and methods in a package

I'm trying to use parallel python in order to do some distributed benchmarking (essentially, coordinate and run some code on a set of machines from a central server). The code I had was working perfectly fine until I moved the functionality to a…
Nikola Knezevic
  • 789
  • 5
  • 20
0
votes
1 answer

How to limit the parallel python code for specific/ required value

I am using parallel Python for running the code below. Could any one correct the code? This should print "SORRY" for the values more than 20 in the inputs list. This is still working for values more than 20. #http://www.parallelpython.com/ import…
HEMS
  • 187
  • 1
  • 5
  • 17
0
votes
2 answers

parallel writing to list in python

I got multiple parallel processes writing into one list in python. My code is: global_list = [] class MyThread(threading.Thread): ... def run(self): results = self.calculate_results() global_list.extend(results) def…
ProfHase85
  • 11,763
  • 7
  • 48
  • 66
0
votes
1 answer

From multiprocessing to distributed processing in python standard library

I am studying this code from gitHub about distributed processing. I would like to thank eliben for this nice post. I have read his explanations but there are some dark spots. As far as I understand, the code is for distributing tasks in multiple…
Thoth
  • 993
  • 12
  • 36
0
votes
1 answer

Error when splitting function argument interval into separate parts. Parallel Python

First off, sorry for all the code. I'm having a hard time formulating a specific question. I have tinkered with this for a while and just can't get it to work. The error message hasn't been very helpful. I would greatly appreciate a little help. I…
tsnitz
  • 15
  • 5
0
votes
1 answer

Parallel Python doesn't give errors but parallel processing doesn't take place

I am new to python and Parallel Python. The problem is I have 4 jobs to be done: generation of 4 masks, multiplication of them with my input image, followed by further processing. Following is the piece of the code written for parallel…
pypro
  • 447
  • 1
  • 7
  • 13
0
votes
2 answers

speedup postgresql to add data from text file using Django python script

I am working with server who's configurations are as: RAM - 56GB Processor - 2.6 GHz x 16 cores How to do parallel processing using shell? How to utilize all the cores of processor? I have to load data from text file which contains millions of…
user2423706
  • 159
  • 4
  • 15
0
votes
1 answer

How to tell parallel-python processes in which folder to search for modules?

I am working on a computer cluster, which has NumPy 1.4.1 installed in the usual folder (/usr/lib64/....). As I want to use NumPy 1.7.0, I have installed it /.../myPath, and added export PYTHONPATH=/.../myPath to my .bashrc, such that using import…
cm_
  • 153
  • 6
1 2 3
4
5