Stack Overflow
Stack Overflow

Questions tagged [process-pool]

For questions related to parallelism achieved through pools of worker processes.

110 questions
0
votes
0 answers

Append to empty list with multiprocessing pool

From a broader level, this is what I am trying to do - append to an empty list an item it reads from the elements list with multiprocessing pool, but this is printing an empty list. Any idea where I am going wrong, or any alternative to achieve the…
user106742
  • 150
  • 1
  • 8
0
votes
0 answers

Sharing a Pool object across processes in python multiprocessing

I am trying to share a pool object between multiple processes using the following code from multiprocessing import Process, Pool import time pool = Pool(5) def print_hello(): time.sleep(1) return "hello" def pipeline(): print("In…
Hardik Aggarwal
  • 49
  • 5
0
votes
1 answer

Python multithreading (Concurrent Futures) resutls in recursive results, how to set multithreading properly?

I put together the following Python script which uses multithreading to execute a function which returns a dictionary (my actual application is for loading and parsing - but simplified it here to a string operation to make it simpler to show). The…
Jose R
  • 930
  • 1
  • 11
  • 22
0
votes
1 answer

Using multiprocessing with a dictionary that needs locked

I am trying to use Python's multiprocessing library to speed up some code I have. I have a dictionary whose values need to be updated based on the result of a loop. The current code looks like this: def get_topic_count(): topics_to_counts = {} …
bballboy8
  • 400
  • 6
  • 25
0
votes
1 answer

How to generate a dataframe from the results of a concurrent.futures.ProcessPoolExecutor()?

My Goal Rewriting a loop (synchronous) using multiprocessing to reduce calculation time (it is looping over millions of rows): def get_info(uid): res = requests.get(CallURL+uid, …
yellow-raven
  • 1
  • 4
0
votes
0 answers

ProcessPoolExecuter with inheretence memory on windows

I want to use ProcessPoolExecuter to decode messages I get from a generator. before initializing the ProcessPoolExecuter I initialize a dictionary needed for the decoding. I want my child processes to copy this dictionary so they can use it to…
user18757515
  • 1
0
votes
1 answer

Using concurrent.futures with infinite iterator and stopping criteria

I am trying to parallelize a loop which uses an infinite generator as input to collect some data and stops when a certain amount of data has been received. My implementation is something like this. class A: def __iter__(self): i = 0 while…
Gerges
  • 6,269
  • 2
  • 22
  • 44
0
votes
1 answer

How to manage the exit of a process without blocking its thread in Python?

I'm trying to code a kind of task manager in Python. It's based on a job queue, the main thread is in charge of adding jobs to this queue. I have made this class to handle the jobs queued, able to limit the number of concurrent processes and handle…
gongar
  • 1
  • 1
0
votes
0 answers

Process Pool Executor runs code outside of scope

I'm trying to run a bunch of processes in parallel with the Process Pool Executor from concurrent futures in Python. The processes are all running in parallel in a while loop which is great, but for some reason the code outside of the main method…
zohani
  • 53
  • 5
0
votes
0 answers

Python proccess pool executor and named tuple leads to unit test error -> "cannot pickle '_io.TextIOWrapper' object"

I am trying to use namedtuple along with process pool executor and when writing a unit test , i keep getting an error below The dut in the code represent the python file where i have the 2 functions stored and i am calling it from the unit…
Nandeep Devendra
  • 91
  • 1
  • 6
0
votes
1 answer

how to properly start parallel executing of two functions over multiple arguments?

I am looking for a way to start two functions in parallel, each executing over a given set of different arguments. I use pool.map to achieve this. I create two different processes and each process starts a pool executing the map. This works - the…
Andre
  • 321
  • 1
  • 12
0
votes
1 answer

Asyncio with ProcessPoolExecutor shutdown before finishing all tasks

I wanted to combine ProcessPoolExecutor with asyncio to run my blocking functions in TestClass concurrently. Each task is intended to be run long-time, so i need a working shutdown process to make things smooth after exiting my script. Any ideas…
hen
  • 65
  • 1
  • 7
0
votes
1 answer

Multiprocessing in a class and use it's instance in another class

I am stuck in using the multiprocessing module of Python. I wanted to create multiprocessing workers in a class and use the object of this class as a composite in another class. Below is the dummy code where the structure looks the same as my…
Abhay Verma
  • 1
  • 4
0
votes
1 answer

Python ProcessPoolExecutor cannot find global variables defined after if __name__ == '__main__'?

The following code will complain NameError: name ‘b’ is not defined from concurrent.futures import ProcessPoolExecutor def add(a): return a+b if __name__ == '__main__': b = 3 with ProcessPoolExecutor() as executor: test =…
Tomoon
  • 91
  • 7
0
votes
0 answers

Multiprocessing ProcessPoolExecutor - BrokenProcessPool Error when run in Windows command line

I am using executor.map and storing results in 'results'. Following code works without any problem on Jupiter notebook. However, it crashes when executing the python script in Windows command prompt. rop_test.py print ('start of process') import…
Ankit Goel
  • 360
  • 1
  • 5
  • 18
1 2 3 4 5
6
7 8