Concerning running parallel code in separate processes (unlike multithreading) and/or related to package `multiprocess` from PyPI.
Questions tagged [multiprocess]
462 questions
11
votes
2 answers
python multiprocessing: no diminishing returns?
Let's say I want to paralelize some intensive computation (not I/O bound).
Naturally, I do not want to run more processes than available processors or I would start paying for context switching (and cache misses).
Mentally, I would expect that as I…
Ricardo Magalhães Cruz
- 3,504
- 6
- 33
- 57
10
votes
4 answers
Python, using multiprocess is slower than not using it
After spending a lot of time trying to wrap my head around multiprocessing I came up with this code which is a benchmark test:
Example 1:
from multiprocessing import Process
class Alter(Process):
def __init__(self, word):
…
Rhys
- 4,926
- 14
- 41
- 64
10
votes
1 answer
Two types of message in a message queue
I was writing a program that starts two processes.
The first process, the "client" sends two type of messages.
The first type increases a shared resource (int).
The second type sets the resource to 0.
After 10 messages, the client has to send a…
EagleOne
- 541
- 1
- 10
- 28
9
votes
2 answers
multiprocessing in python - what gets inherited by forkserver process from parent process?
I am trying to use forkserver and I encountered NameError: name 'xxx' is not defined in worker processes.
I am using Python 3.6.4, but the documentation should be the same, from…
sgyzetrov
- 191
- 1
- 9
9
votes
1 answer
matplotlib error when running plotting in multiprocess
I am using python's Multiprocess.Pool to plot some data using multiple processes as follows:
class plotDriver:
def plot(self, parameterList):
numberOfWorkers = len(parameterList)
pool = Pool(numberOfWorkers)
…
Yingrui Chang
- 139
- 5
9
votes
3 answers
Performance difference for multi-thread and multi-process
A few years ago, in the Windows environment, I did some testing, by letting multiple instances of CPU computation intensive + memory access intensive + I/O access intensive application run. I developed 2 versions: One is running under…
Cheok Yan Cheng
- 47,586
- 132
- 466
- 875
8
votes
1 answer
python: How to debug multiprocess? (using eclipse+pydev)
I've seen a couple of questions on the topic but I didn't get a full answer...
My code is basically:
from multiprocessing import Process
p = Process(target=f).start()
p.join()
def f():
print 'break!'
And I want to put a breakpoint on the print.…
Guy
- 14,178
- 27
- 67
- 88
8
votes
2 answers
How to use multiprocess in python on a class object
I am fairly new to Python, and my experience is specific to its use in Powerflow modelling through the API provided in Siemens PSS/e. I have a script that I have been using for several years that runs some simulation on a large data set.
In order…
Josh
- 81
- 1
- 2
8
votes
1 answer
Preferred way to empty multiprocessing.queue(-1) in python
I want to pull all the items currently in a queue. There is another thread constantly putting items in the other end, and every period I want to get all the items currently in the queue.
Is there some reason to prefer:
res = []
while q.qsize > 0 :
…
phil_20686
- 4,000
- 21
- 38
8
votes
1 answer
Multiprocess sqlite INSERT: "database is locked"
(Please note: There is a question called "SQLite3 and Multiprocessing" but that question is actually about multithreading and so is the accepted answer, this isn't a duplicate)
I'm implementing a multiprocess script, each process will need to write…
Juicy
- 11,840
- 35
- 123
- 212
8
votes
1 answer
Python Process blocked by urllib2
I set up a process that read a queue for incoming urls to download but when urllib2 open a connection the system hangs.
import urllib2, multiprocessing
from threading import Thread
from Queue import Queue
from multiprocessing import Queue as…
Davide Muzzarelli
- 929
- 1
- 9
- 13
7
votes
1 answer
Python, multithreading too slow, multiprocess
I'm a multiprocessing newbie,
I know something about threading but I need to increase the speed of this calculation, hopefully with multiprocessing:
Example Description: sends string to a thread, alters string + benchmark test,
send result back…
Rhys
- 4,926
- 14
- 41
- 64
7
votes
1 answer
Python multiprocess.Pool.map can't handle large arrays.
This is the code I use to parrellize an apply function on the lines of a pandas.DataFrame object:
from multiprocessing import cpu_count, Pool
from functools import partial
def parallel_applymap_df(df: DataFrame, func,…
Giovanni Barbarani
- 239
- 1
- 11
7
votes
3 answers
Deadlock with logging multiprocess/multithread python script
I am facing the problem with collecting logs from the following script.
Once I set up the SLEEP_TIME to too "small" value, the LoggingThread
threads somehow block the logging module. The script freeze on logging request
in the action function. If…
Jiří Polcar
- 1,192
- 9
- 15
7
votes
1 answer
Multiprocess multiple files in a list
I am trying to read a list that contains N number of .csv files stored in a list synchronously.
Right now I do the following:
import multiprocess
Empty list
Append list with listdir of .csv's
def A() -- even files (list[::2])
def B() -- odd files…
Christopher W
- 139
- 2
- 2
- 8