Stack Overflow
Stack Overflow

Questions tagged [parallel-python]

Parallel Python provides mechanism for parallel execution of python code on SMP (systems with multiple processors or cores) and clusters (computers connected via network)

Parallel Python (PP) is a Python module which provides mechanism for parallel execution of python code on SMP (systems with multiple processors or cores) and clusters (computers connected via network).

PP module overcomes the limitations imposed by the GIL (Global Interpreter Lock) on multithreading and provides a simple way to write parallelPpython applications. Internally ppsmp uses processes and IPC (Inter Process Communications) to organize parallel computations.

All the details and complexity of the latter are completely taken care of, and your application just submits jobs and retrieves their results (the easiest way to write parallel applications).

The software written with PP works in parallel even on many computers connected via local network or Internet (clusters). Cross-platform portability and dynamic load-balancing allows PP to parallelize computations efficiently even on heterogeneous and multi-platform clusters.

65 questions
1
vote
2 answers

Parallel Python: 4 threads have same speed as 2 threads

I'm using Parallel Python for executing a computation heavy code on multiple cores. I have an i7-4600M processor, which has 2 cores and 4 threads. The interesting thing is, the computation takes nearly the same time if I use 2 or 4 theads. I wrote…
Jakube
  • 3,353
  • 3
  • 23
  • 40
1
vote
2 answers

local variable 'sresult' referenced before assignment

I have had multiple problems trying to use PP. I am running python2.6 and pp 1.6.0 rc3. Using the following test code: import pp nodes=('mosura02','mosura03','mosura04','mosura05','mosura06', 'mosura09','mosura10','mosura11','mosura12') def…
Wolfgang Kerzendorf
  • 724
  • 1
  • 9
  • 24
1
vote
0 answers

How can I resolve the "import multiarray" error in parallel python?

I am trying to run a python 2.7 script which uses parallel python (version 1.6.1) to excute a function which uses numpy arrays (numpy version 1.6.1) on a Ubuntu Voyager (Ubuntu 12.04 derivative) system. It gives me the following error message…
user3734171
  • 11
  • 1
1
vote
0 answers

parallel python timeout spawns another process instead of failing?

Using parallel python 1.6.4 I spawn a subprocess.Popen command on a remote server. For whatever reason, the command isn't completing in a timely matter, i.e., within the socket_timeout I've set. In this case, I expected parallel python to fail,…
CAB
  • 1,015
  • 1
  • 14
  • 24
1
vote
0 answers

Parallel Python module using other server

I'm using parallel python to do some number crunching on several computers. Everything was working fine till I thought it would be nice to include some classes. Here is the code for submitting the jobs: for i in xrange(n_cpus): …
SirJohnFranklin
  • 57
  • 4
1
vote
2 answers

How to dynamically scale StarCluster/qsub/EC2 to run parallel jobs across multiple nodes

I am new to using Startcluster/qsub/grid engine to run parallel jobs and I tried reading couple of other posts regarding the same. I still am not sure of how to build a scalable solution for my specific requirement. I would like to take in some more…
user1652054
  • 445
  • 2
  • 11
  • 23
1
vote
2 answers

Too Many Open Files Parallel Python Subprocess error

Questions with similar issue: Parallel Python - too many files and Python too many open files (subprocesses) I am using Parallel Python [V1.6.2] to run tasks. The task processes an input file and outputs a log/report. Say, there are 10 folders each…
user1652054
  • 445
  • 2
  • 11
  • 23
1
vote
1 answer

Caching error in parallel python

I am having a problem with the transparent caching when using the parallel python module. I changed a method signature in a module called by my parallelized code. When I execute the code it gives me an error saying it expected the old method…
jonathanbyrn
  • 719
  • 6
  • 13
1
vote
2 answers

Updating database with callback in Parallel Python

I'm trying to do some text processing on around 200,000 entries in a SQlite database which I'm accessing using SQLAlchemy. I'd like to parallelize it (I'm looking at Parallel Python), but I'm not sure how exactly to do it. I want to commit the…
abroekhof
  • 796
  • 1
  • 7
  • 20
1
vote
0 answers

Parallel python worker graceful failure

While using pp to parallelize a significantly complex machine learning problem I'm finding myself having to rely fairly extensively on third party libraries which are of varying quality. One in particular has a decent amount of edge case crashes…
amirpc
  • 1,638
  • 3
  • 19
  • 24
0
votes
0 answers

Adding elements of first pandas df in a new column of the second pandas df with a condition

I wrote a python script with a nested for loop which adds elements of first pandas df in a new column of the second pandas df with a condition that element of a column in first pandas df is in between ANY elements of two columns of the second…
Mrinal Mishra
  • 25
  • 3
0
votes
1 answer

Opening multiple Websockets in parallel in Python

I am using the websocket library in Python and I am new to this. I want to create multiple different connections to websockets. This happens through my custom WebsocketProcess class which opens the connection, receives the event, keeps a record Id…
Erunaven
  • 5
  • 3
0
votes
1 answer

parallel python not discovering worker nodes

I am trying to implement a basic pp example as shown below (SCRIPT1). import pp ppservers = ("*",) # autodiscovery mode on! # create the job server job_server = pp.Server(ppservers=ppservers) for computer, cpu_count in…
dreamer
  • 901
  • 2
  • 15
  • 38
0
votes
1 answer

Parallel Python not seeing all computers

I am trying to use Parallel Python with Python 3.5 on Windows 10. I'm new to this, so please excuse the terminology. I have installed Python and all of the required packages on each of the computers (nodes) and have been running a batch script on…
jlt199
  • 2,349
  • 6
  • 23
  • 43
0
votes
1 answer

how to call a module in a function in pp where that fuction has other functions in it?

I'm currently using parallel python ,and in the parameters of job_server.submit i added the library in modules but the problem is that even that library has other librairies in it .so what should i do ? Here is the code i'm trying to run: from…
KAFFEL
  • 35
  • 1
  • 8
1 2
3
4 5