For questions related to parallelism achieved through pools of worker processes.
Questions tagged [process-pool]
110 questions
3
votes
1 answer
multiprocessing.Pool: How to start new processes as old ones finish?
I'm using multiprocessing Pool to manage tesseract processes (OCRing pages of microfilm). Very often in a Pool of say 20 tesseract processes a few pages will be more difficult to OCR, and thus these processes are taking much much longer than the…
Tristan
- 1,730
- 3
- 20
- 25
3
votes
1 answer
Python multiprocessing: abort map on first child error
What's the proper way of aborting multiprocessing when one of the child aborts and/or throw an Exception?
I found various questions around that (generic multiprocessing error handling, how to close multiprocessing pool on exception but without…
bagage
- 1,094
- 1
- 21
- 44
3
votes
2 answers
class attributes and memory shared between Processes in process pool?
I have a class A that when initiated changes a mutable class attribute nums.
when initiating the class via a Process pool with maxtasksperchild= 1, I notice that nums has the values of several different processes. which is an undesirable behavior…
moshevi
- 4,999
- 5
- 33
- 50
3
votes
0 answers
Using multiprocessing and ProcessPoolExecutor simultaneously
I am trying to create a simple script for python3.5 that can execute heavy computer vision algorithms in parallel. I have created a process by multiprocessing.Process in main process.
Inside that process I create…
user3398248
- 31
- 1
2
votes
1 answer
How to use Queue correctly with ProcessPoolExecutor in Python?
I am trying to use a queue to load up a bunch of tasks and then have a process pool setup go at it where each process pops-out a task out of the queue and works on it. I am running into problems in that the setup is not working. Something is…
Buddy Li
- 35
- 5
2
votes
1 answer
Large dataset, ProcessPoolExecutor issues
PROBLEM - ProcessPoolExecutor hasn't increased speed. Confirmed by tqdm
Learned enough about python to copy and/or write a program that works. each file takes ~40 seconds to load->filter->write. I have ~6,800 files to work through and want a better…
Yusuf
- 23
- 4
2
votes
2 answers
multiprocessing.Pool map multiple arguments with shared value (Resolved)
I am practicing on using shared values for multiprocessing. I have an existing Process function that is working using shared value:
def run_procs_with_loop(lock):
# this is my shared value
shared_number = Value('i', 0)
…
punsoca
- 459
- 1
- 7
- 15
2
votes
1 answer
Python concurrent.futures.ProcessPoolExecutor: Lot of RAM for large number of tasks
I am using concurrent.futures.ProcessPoolExecutor to run python codes in parallel. Basically what I do is
with concurrent.futures.ProcessPollExecutor(max_workers=10) as executor:
futures = {executor.submit(my_function, i)
for i in…
kvng vikram
- 21
- 4
2
votes
0 answers
Broken Process Pool error using fbProphet cross validation
I'm using facebook's Prophet library to do a forecast, but before I put it in production I want to validate it using the included cross validation package. I have a machine with 64GB RAM and Ryzen 9 5950X. I get this error: "BrokenProcessPool: A…
XiB
- 620
- 6
- 19
2
votes
0 answers
Handling interrupts in asyncio jobs running in separate processes with done callbacks
I'm running a large number of CPU-bound computations in separate processes and want to perform some callback as each job finishes. To do this I'm combining asyncio with a concurrent.futures.ProcessPoolExecutor, and I'm setting a callback on each…
Sean
- 1,346
- 13
- 24
2
votes
1 answer
How to pass 2d array as multiprocessing.Array to multiprocessing.Pool?
My aim is to pass a parent array to mp.Pool and fill it with 2s while distributing it to different processes. This works for arrays of 1 dimension:
import numpy as np
import multiprocessing as mp
import itertools
def worker_function(i=None):
…
Artur Müller Romanov
- 4,417
- 10
- 73
- 132
2
votes
2 answers
How to assign values to array from inside the worker_funtion of multiprocessing.Pool.map?
Basically what I want is to insert those 2's into ar, so that ar gets changed outside the worker_function.
import numpy as np
import multiprocessing as mp
from functools import partial
def worker_function(i=None, ar=None):
val = 2
ar[i] =…
Artur Müller Romanov
- 4,417
- 10
- 73
- 132
2
votes
1 answer
Killing processes in ProcessPoolExecutor
I am using Python's ProcessPoolExecutor to run multiple processes in parallel and process them as any of them finishes. Then I look at their output and as soon as at least one of them gives satisfying answer I want to exit the program.
However, this…
user1635881
- 239
- 3
- 11
2
votes
1 answer
Update variable while working with ProcessPoolExecutor
if __name__ == '__main__':
MATCH_ID = str(doc_ref2.id)
MATCH_ID_TEAM = doc_ref3.id
with concurrent.futures.ProcessPoolExecutor(max_workers=30) as executor:
results = list(executor.map(ESPNlayerFree, teamList1))
…
johnrao07
- 6,690
- 4
- 32
- 55
2
votes
1 answer
Python concurrent.futures using subprocess, running several python script
I want to run several python script at the same time using concurrent.futures.
The serial version of my code go and look for a specific python file in folder and execute it.
import re
import os
import glob
import re
from glob import glob
import…
Angel_M
- 97
- 9