For questions related to parallelism achieved through pools of worker processes.
Questions tagged [process-pool]
110 questions
1
vote
1 answer
Why the performance of concurrent.futures.ProcessPoolExecutor is very low?
I'm trying to leverage concurrent.futures.ProcessPoolExecutor in Python3 to process a large matrix in parallel. The general structure of the code is:
class X(object):
self.matrix
def f(self, i, row_i):
def…
AmirHJ
- 827
- 1
- 11
- 21
0
votes
1 answer
Python method with ProcessPoolExecutor freeze QT gui
One of buttons in my qt gui starts ProcessPoolExecutor which is making tifs images.
I want add process bar but my gui is freezing on time when the event loop is working.
@pyqtSlot()
def on_pb_start_clicked(self):
if…
Mily
- 3
- 1
0
votes
0 answers
BrokenProcessPool error - Help me figure out why
Re-post to fix a couple of typographical errors.
I'm trying to work through some training videos for asynchronous processing.
My environment is a local Jupyter Notebook with the necessary modules installed.
Here's the code
from concurrent.futures…
THill3
- 87
- 6
0
votes
0 answers
python grpc module has a process issue use with subprocess.getstatusoutput
Project introduction:
run in python env, grpc 1.38;
use flask frame and etcd3, flask frame use to accept the request, and use etcd3 to search the etcd datebase.
When the process problem appear:
while the flask frame accept a request, than deal…
HEtinto
- 1
- 1
0
votes
1 answer
How to pass variables to ProcessPoolExecutor, got type error
i'm trying to modify my code to apply Parallelism with ProcessPoolExecutor. as i read on internet i should use Map() function to pass my variables to the function. but it throws typeerror, which needs iterabels. should i modify the method so it…
Mostafa Bouzari
- 9,207
- 3
- 16
- 26
0
votes
0 answers
issue regarding concurrent.futures.ProcessPoolExecutor
I am learning Multiprocessing in order to use it in project. I can not understand why ProcessPoolExecutor is taking so much time.
Am I doing something wrong in execution or it shouldn't be implemented in such cases.
Please help me out.
import…
Tabish Izhar
- 47
- 3
0
votes
0 answers
concurrent.futures.processpoolexecutor only using two threads at once
Consider the following code which used concurrent.futures.ProcessPoolExecutor() to run a set of test code.
import time
start =time.time()
def test_function(x):
y=0;
for ix in range (0,x):
y=y+ix*2
time.sleep(1);
return…
ShoutOutAndCalculate
- 457
- 5
- 16
0
votes
1 answer
In Python ProcessPoolExecutor, do you need call shutdown after getting a BrokenProcessPool exception?
In Python ProcessPoolExecutor, do you need call shutdown after getting a BrokenProcessPool exception?
Say I have something like this:
pool = ProcessPoolExecutor(max_workers=1)
try:
return pool.submit(
do_something,
…
cozos
- 787
- 10
- 19
0
votes
1 answer
Working with ProcessPoolExecutor.(Windows) Where do all the classes and functions go, that have nothing to do with multiprocessing?
+++ I EDITED MY QUESTION, TO REFLECT WHAT I THINK IS THE CORRECT STRUCTURE NOW, so that not necessary things don't get loaded in the child workers.. with the help of @booboo +++ THX
Where do all my classes and functions go(that have nothing to do…
Leonick
- 45
- 7
0
votes
1 answer
How to properly transform a sync function to an async one?
I'm writing a telegram bot and I need the bot to be available to users even when it is processing some previous request. My bot downloads some videos and compresses them if it exceeds the size limit, so it takes some time to process the request. I…
Mikhail
- 132
- 7
0
votes
1 answer
concurrent.futures ProcessPoolExecutor restarts script for every Process
I am currently experimenting with ProcessPoolExecutor instead of Threadpools, because i need an alternative that i can cancel when a network request doesn't answer.
Now, i stumbled upon a strange side effect of using the processpoolexecutor:
For…
PyNoob
- 9
- 3
0
votes
0 answers
ProcessPoolExecutor stuck indefinitely when child process dies
I have a script running on one of my linux servers which handles batch file processing with a ProcessPoolExecutor and generally runs fine days or even weeks on end without any issue. Sometimes though it looks like a few of my child processes just…
zeitghaist
- 89
- 9
0
votes
1 answer
some problem when use ProcessPoolExecutor to split data into smaller piece
I am trying to split a large file into smaller pieces. I will read all the data from the large file first and then use multiprocessing to write them to different smaller data file.
Here is method one, where is use multiprocessing.Process to…
pythonHua
- 61
- 6
0
votes
1 answer
Multiprocessing using Functions Held in Variables as the Called Function
If I have this:
import concurrent.futures
def SetParameters():
return { "processVal" : 'q',
"keys" : { "key1" : { "fn" : a,
"future" : None,
"futureResult" :…
Weylin Piegorsch
- 74
- 7
0
votes
0 answers
ProcessPoolExecutor doesnt kill forked python process even after its completion
I am creating ProcessPoolExecutor and then submitting some tasks to be processed. ProcessPoolExecutor forks multiple processes to do the task. The issue I am facing is some of the forked processes are stuck with their tasks even after the…
Programmer
- 165
- 1
- 1
- 7