For questions related to parallelism achieved through pools of worker processes.
Questions tagged [process-pool]
110 questions
2
votes
1 answer
Parallel processing with ProcessPoolExecutor
I have a huge list of elements which somehow must be processed.
I know that it can be done with Process from multiprocessing by:
pr1 = Process(calculation_function, (args, ))
pr1.start()
pr1.join()
and so I can create lets say 10 processes and…
John
- 451
- 6
- 21
2
votes
1 answer
multiprocessing pool not working in nested functions
Following code not executing as expected.
import multiprocessing
lock = multiprocessing.Lock()
def dummy():
def log_results_l1(results):
lock.acquire()
print("Writing results", results)
lock.release()
def…
kanna
- 1,412
- 1
- 15
- 33
1
vote
0 answers
ProcessPoolExecutor hangs
I'm trying to use ProcessPoolExecutor with the following function:
def get_ellipse_properties(a,b,mask=mask):
''' a - int time frame value of a feature
b - int feature id number
mask- iris cube with segmentation mask'''
…
yuliya
- 55
- 5
1
vote
0 answers
parallism python using multiprocessing
I have been trying to use multiprocessing module from python to achieve parallism.
I'm able to execute my code, it run in parallel but after some time just only one process finishes its task and the others exit without finishing I know there is…
HINF
- 11
- 4
1
vote
2 answers
best way to speed up multiprocessing code in python?
I am trying to mess around with matrices in python, and wanted to use multiprocessing to processes each row separately for a math operation, I have posted a minimal reproducible sample below, but keep in mind that for my actual code I do in-fact…
Chris
- 95
- 8
1
vote
0 answers
How can I debug a python code that contains Multiprocesses and Multi threads?
I am writing a python code that contains the use of ThreadPoolExecutor and ProcessPoolExecutor.
I would like to debug the code, specially the functions that are being called in parallel. Being able to stop execution, set breakpoints, inspect…
KansaiRobot
- 7,564
- 11
- 71
- 150
1
vote
2 answers
python concurrent.futures.ProcessPoolExecutor crashing with full RAM
Python concurrent.futures.ProcessPoolExecutor crashing with full RAM
Program description
Hi, I've got a computationally heavy function which I want to run in parallel. The function is a test that accepts as inputs:
a DataFrame to test on
parameters…
janboro
- 79
- 7
1
vote
1 answer
Python: Multiprocessing with pool.map while main is still working
I am a few days into learning python, and would like to understand this.
I am doing a file explorer, and want to speed up thumbnail creation.
Watched a bunch of tutorials about multiprocessing, but none show hot to continue main(), while processes…
Leonick
- 45
- 7
1
vote
1 answer
Increase in python scripts execution time when using ProcessPoolExecutor
I am observing increase in execution time of python script when I trigger parallel instances of it using process pool executor on a 56 core machine. The script abc.py imports a heavy python library which takes around 1 seconds.
time python…
Deepanshu Arora
- 375
- 1
- 5
- 21
1
vote
0 answers
How to manage memory usage of processes in Linux?
I am running a program that is effectively a server, where you enter a job request, the server computes a heavy calculation in a process (using a ProcessPoolExecutor) and then saves the result on the server, which can be retrieved by the user.
Some…
Tom McLean
- 5,583
- 1
- 11
- 36
1
vote
1 answer
parallelizing code with python pathos multiprocessing in docker
I am parallizing code across 30 CPUs and confirmed that outside a container this works fine using the python library 'pathos'.
pool = ProcessPool(nodes=30)
results = pool.map(func_that_needs_to_run_in_parallel,…
hoola_huupsh
- 11
- 4
1
vote
1 answer
How to run multiple asyncio loops inside syncrhonous sub-processes inside a main asyncio loop?
I have a main function which I run with asyncio and then inside it I used event_loop.run_in_executor() to run mutliple processes of a blocking function.
What I wish to do is inside each of these processes to run a new asyncio loop for each one in…
KZiovas
- 3,491
- 3
- 26
- 47
1
vote
1 answer
asyncio: can a task only start when previous task reach a pre-defined stage?
I am starting with asyncio that I wish to apply to following problem:
Data is split in chunks.
A chunk is 1st compressed.
Then the compressed chunk is written in the file.
A single file is used for all chunks, so I need to process them one by…
pierre_j
- 895
- 2
- 11
- 26
1
vote
1 answer
ProcessPoolExectur and Ctrl C
I'm using ProcessPoolExecutor on Windows 10. Python version is 3.9.5.
When I press Ctrl+C twice the program hangs endless, even if I set a timeout.
with concurrent.futures.ProcessPoolExecutor() as executor:
results = executor.map(Worker,…
Martin Bammer
- 537
- 7
- 19
1
vote
0 answers
ModuleNotFoundError when importing functions and using ProcessPoolExecutor
I am trying to execute a python script (here called file_2.py) in parallel using ProcessPoolExecutor. The script uses functions I have written in another file (here called file_1.py). Both of these files are in the same directory.
If I important the…
iClemens
- 11
- 1