For questions related to parallelism achieved through pools of worker processes.
Questions tagged [process-pool]
110 questions
0
votes
1 answer
How to use Process Pool Executor in tornado with synchronous code?
I am new to tornado and have an API that makes a blocking database call. Because of this blocking call, tornado isn't able to serve all the requests if multiple requests come at the same time.
I looked about it and found out that it could be solved…
ATK
- 358
- 3
- 17
0
votes
0 answers
How to complete a childprocess's job when the parent is interrupted?
I run the following code, and I want the child process to continue to complete the job when the parent process is terminated.
I need the child process to complete each submitted job.
The parent process doesn't matter.
import signal
from…
user15968489
0
votes
1 answer
I am having problems with ProcessPoolExecutor from concurrent.futures
I have a big code that take a while to make calculation, I have decided to learn about multithreading and multiprocessing because only 20% of my processor was being used to make the calculation. After not having any improvement with multithreading,…
nobody48sheldor
- 30
- 10
0
votes
0 answers
Multiprocessing pool.imap with large chunksize is skipping processing some records in iterable
I am processing a list of string using multiprocessing pool.imap() with passing chunksize. My list length is 1821 and process in imap is 4. I was trying to give almost equal number of chunk size to each process so set chunk size as 455. Also tried…
Garima
- 11
- 2
0
votes
1 answer
Show the running executors identifications in ProcessPoolExecutor
This is the problem I am having. I wont share code because of condfidentiality but instead I will provide some dummy example.
Assume that we have a class as follows:
class SayHello:
def __init__(self, name, id):
self.name=name
…
Payam30
- 689
- 1
- 5
- 20
0
votes
1 answer
Python Concurrent.futures CPU usage query
When using concurrent.futures in Python for large data sets ( 9x200,000x4 np.floats ) I've noticed that CPU usage is low (13% ~ equivalent to 1 core being used) for the beginning. However after a while it shoots up to what I expect for…
ARLI
- 1
0
votes
1 answer
Tornado ProcessPoolExecutor: how to limit the number of processes
I am writing a REST server in Tornado. I use a ProcessPoolExecutor with a max_workers configurable parameter.
However, the problem is that it seems to not effectively limit the number of processes the way I want.
The code is
def post(self):
…
Lore
- 1,286
- 1
- 22
- 57
0
votes
1 answer
BrokenProcessPool error while trying to run docplex example
I am trying to run some cplex models in parallel with python process pool. I tried to run this as an example of process pool with docplex on my windows 10 -spyder 3.6.9.
When I run I get this error :
File…
Sana.Nz
- 81
- 11
0
votes
0 answers
How to solve the PicklingError produced by ProcessPoolExecutor applied on a single function, while ThreadPoolExecutor works fine [concurrent.futures]
I would like to run concurrently a simple function that writes the output of a process into a txt.file and then stores it to DBFS (Databricks filesystem). In my example I use both the ThreadPoolExecutor class() and the ProcessPoolExecutor class()…
NikSp
- 1,262
- 2
- 19
- 42
0
votes
1 answer
Wondering if you could have multiple functions in a task for process pool python
def task():
print("Executing our Task on Process {}".format(os.getpid()))
def main():
executor = ProcessPoolExecutor(max_workers=3)
task1 = executor.submit(task)
task2 = executor.submit(task)
if __name__ == '__main__':
…
0
votes
0 answers
any code with multiprocessing in python not working
I have a simple multiprocessing code
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
This code gives same error repeated over(as given below) . Not…
Prateek Agarwal
- 59
- 1
- 6
0
votes
1 answer
How to submit `Value` instance to `ProcessPoolExecutor`?
I want to use sharing state with ProcessPoolExecutor
Code:
from multiprocessing import Value
from concurrent.futures import ProcessPoolExecutor
def function(times, a):
print('I\'m here')
for _ in range(times):
with a.get_lock():
…
Pavel Antspovich
- 1,111
- 1
- 11
- 27
0
votes
2 answers
`multiprocessing.Pool.map()` seems to schedule wrongly
I have a function which request a server, retrieves some data, process it and saves a csv file. This function should be launch 20k times. Each execution last differently: sometimes It last more than 20 minutes and other less than a second. I decided…
lsmor
- 4,698
- 17
- 38
0
votes
1 answer
How to properly memoize when using a ProcessPoolExecutor?
I suspect that something like:
@memoize
def foo():
return something_expensive
def main():
with ProcessPoolExecutor(10) as pool:
futures = {pool.submit(foo, arg): arg for arg in args}
for future in…
GL2014
- 6,016
- 4
- 15
- 22
0
votes
1 answer
How can I use a shared/managed dictionary with a process pool (Python 3.x)
I am working on a project that requires me to extract a ton of information from some files. The format and most of the information about the project does not matter for what I am about to ask. I mostly do not understand how I would share this…
Paul
- 11
- 1