For questions related to parallelism achieved through pools of worker processes.
Questions tagged [process-pool]
110 questions
0
votes
0 answers
Synchronize ProcessPoolExecutor
I am trying to run some very large time-series data using concurrent.futures.ProcessPoolExecutor(). The dataset contains multiple time series (that are independent). The entire dataset is available in a list of tuples data that I pass through a…
jamie
- 89
- 4
- 11
0
votes
1 answer
Multiprocessing Pool apply_async's workers and callback aren't called when executing in the inner scope
Here is the code I have:
import pandas as pd
import multiprocessing as mp
CPU = 4
inp = pd.DataFrame({ 'col': ['a', 'b'] })
def test(dataframe):
df = dataframe.copy()
def worker(data):
print('worker')
def callback(data):
…
eawer
- 1,398
- 3
- 13
- 25
-1
votes
0 answers
BrokenProcessPool error seems incompatible with other debugging statements
Trying to work through some training videos for asynchronous processing.
Using a local Jupyter Notebook with the necessary modules installed.
Here's the code
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures import…
THill3
- 87
- 6
-1
votes
1 answer
Using ProccessPoolExecutor for functions with I/O
Lately I have been using ProcessPoolExecutor for accelerating the processing of some functions I wrote.
I have a question regarding one function I would like to accelerate.
This function
def thefunction(input_file, output_file, somepar)
Involves…
KansaiRobot
- 7,564
- 11
- 71
- 150
-1
votes
1 answer
is it possible to create process pool in golang
we have a async task scheduler system, which use golang's exec.Command to execute php scripts. Each time, the scheduler fetch a task from message queue, it will create a new exec.Command to execute the task.
Sometimes, there are almost thousands of…
shimron
- 596
- 6
- 19