Stack Overflow
Stack Overflow

Questions tagged [python-multiprocessing]

multiprocessing is a package that supports spawning processes using an API similar to the threading module in python programming language. When asking a question about this topic, please mention the operating system you are running it on, or add it to the tags.

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using Operating System processes instead of threads.

The multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

4758 questions
1
vote
2 answers

Can I dynamically register objects to proxy with a multiprocessing BaseManager?

There are plenty of examples of using a multiprocessing BaseManager-derived class to register a method for returning a queue handle proxy, that clients can then use to pull/put from the queue. This is great, but I have a different scenario - what if…
John Allard
  • 3,564
  • 5
  • 23
  • 42
1
vote
1 answer

Create dict in one module and access it in another module through multiprocessing

I am new to python and multiprocessing concepts in python (this is my first python project). I have written few modules and wired them up together to work in sequential manner. Right now, i have requirement to fasten few things. What i want to…
Strive
  • 43
  • 4
1
vote
1 answer

Is my understanding of multiprocessing's starmap correct?

I wrote a little program to check a bunch of hashes, and I was wondering if this is a correct application of starmap and if it should be making other processes. My current understanding is that starmap is supposed to divide the task amongst the…
NeffPeff
  • 46
  • 5
1
vote
1 answer

How to handle errors in multiprocesses?

MWE is below. My code spawn processes with torch.multiprocessing.Pool and I manage the communication with the parent with a JoinableQueue. I followed some online guide to handle gracefully CTRL+C. Everything works fine. In some cases (my code has…
Simon
  • 5,070
  • 5
  • 33
  • 59
1
vote
2 answers

Using multiprocessing to double the speed of working on a list

Let's say I have a list like this: list_base = ['a','b','c','d'] If I used for xxx in list_base:, the loop would parse the list one value at a time. If I want to double the speed of this work, I'm creating a list with two values to iterate over at…
Digital Farmer
  • 1,705
  • 5
  • 17
  • 67
1
vote
0 answers

how do i run a telethon async function in multiprocessing in python

Am trying to run an async telethon function under multiprocessing but i don't know how to blend the two together. i have Googled, search and ask but it seems like there is no topic like this on the internet import multiprocessing as…
kruz
  • 21
  • 3
1
vote
2 answers

Race Condition in Python Multiprocessing Aync_apply

Could there be a race condition in the following code such that two async processes both try to update the global variable B at the same time in the call back function? If so, does python handle this or is it something we have to handle using locks?…
indispinablenorm
  • 55
  • 5
1
vote
0 answers

Multiprocessing in Python pandas for 300k rows

We have data of 300 000 rows representing workers balance histories. The goal is to add special feature counting number of refunding of penalties (which can detected by sum 500 or 1000 in amount column). Original "for" cycle seeks 20 hours to finish…
Nikita Tsekhanovich
  • 119
  • 2
  • 10
1
vote
1 answer

Using multiprocessing and GNU Parallel at the same time?

I have a Python script that is currently using multiprocessing to perform tasks in parallel. My advisor recommended me to use GNU Parallel to speed it up since Python programs always execute on a single core. Should I keep the multiprocessing script…
raffayatiq
  • 159
  • 9
1
vote
1 answer

How to properly setup nested proxy variables with BaseManager?

Documentation mentions that you're able to setup a Manager with nested proxy variables, but I am unable to find any examples or get this to work. I am using flask which runs through an init process, which the below code snippet is from. Each PID…
Kyle Smith
  • 43
  • 5
1
vote
2 answers

Weird behaviour of Queue.empty in Python

I came across this weird issue with multiprocessing's Queue.empty() in Python. The following code output is True and 20, right after filling it with elements. from multiprocessing import Queue import random q = Queue() for _ in range(20): …
balu
  • 111
  • 1
  • 8
1
vote
1 answer

Python Multiprocessing Processes Go 100% to zero without finishing

I am running Jupiter notebook to process some huge load on a list of arrays as: import multiprocessing as mp print("Number of processors: ", mp.cpu_count()) pool = mp.Pool(mp.cpu_count()) try: a_results = pool.map_async(process_each_area, [area…
sandeepsign
  • 539
  • 6
  • 11
1
vote
0 answers

get contiguous numbers for workers in a `multiprocessing.Pool`

GPU models are often ran efficiently by parallelization over batches or features or other tensor dimensions, and theoretically don't need multiprocessing over one device. Nonetheless, from a (developer) productivity point of view it could become…
Herbert
  • 5,279
  • 5
  • 44
  • 69
1
vote
0 answers

How can a tmpfs folder in Docker be mounted and shared on Windows?

My application is utilizing python multiprocessing.shared_memory. On Linux, a tmpfs folder is created as the /dev/shm folder for shared_memory. In order to share memory between processes and containers, those other processes and containers would…
geekgeek4
  • 105
  • 10
1
vote
0 answers

Python multiprocessing execute a function before Pool closes

I'm using python's multiprocessing module for parallelising and processing my data. Before I close this process pool, I want to execute a function for each of the sub-processes that are spawned. How can I achieve this? eg, below is a sample…
vish4071
  • 5,135
  • 4
  • 35
  • 65
1 2 3
…
99
100