Stack Overflow
Stack Overflow

Questions tagged [python-multiprocessing]

multiprocessing is a package that supports spawning processes using an API similar to the threading module in python programming language. When asking a question about this topic, please mention the operating system you are running it on, or add it to the tags.

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using Operating System processes instead of threads.

The multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

4758 questions
47
votes
4 answers

Starmap combined with tqdm?

I am doing some parallel processing, as follows: with mp.Pool(8) as tmpPool: results = tmpPool.starmap(my_function, inputs) where inputs look like: [(1,0.2312),(5,0.52) ...] i.e., tuples of an int and a float. The code runs nicely, yet…
sdgaw erzswer
  • 2,182
  • 2
  • 26
  • 45
45
votes
3 answers

Keras + Tensorflow and Multiprocessing in Python

I'm using Keras with Tensorflow as backend. I am trying to save a model in my main process and then load/run (i.e. call model.predict) within another process. I'm currently just trying the naive approach from the docs to save/load the model:…
John Cast
  • 1,771
  • 3
  • 18
  • 40
43
votes
1 answer

Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python

Multiprocessing is a powerful tool in python, and I want to understand it more in depth. I want to know when to use regular Locks and Queues and when to use a multiprocessing Manager to share these among all processes. I came up with the following…
SmCaterpillar
  • 6,683
  • 7
  • 42
  • 70
42
votes
4 answers

How to solve memory issues while multiprocessing using Pool.map()?

I have written the program (below) to: read a huge text file as pandas dataframe then groupby using a specific column value to split the data and store as list of dataframes. then pipe the data to multiprocess Pool.map() to process each dataframe…
everestial007
  • 6,665
  • 7
  • 32
  • 72
41
votes
5 answers

Python Multiprocessing error: AttributeError: module '__main__' has no attribute '__spec__'

I'm using Python 3.6 and am trying to follow along with the very first example at the website below (full code also below) and am getting the below error: https://docs.python.org/3.6/library/multiprocessing.html Error message: AttributeError: module…
user8474060
  • 809
  • 1
  • 7
  • 7
38
votes
3 answers

Multiprocessing: use only the physical cores?

I have a function foo which consumes a lot of memory and which I would like to run several instances of in parallel. Suppose I have a CPU with 4 physical cores, each with two logical cores. My system has enough memory to accommodate 4 instances of…
user189035
  • 5,589
  • 13
  • 52
  • 112
35
votes
4 answers

Does python logging support multiprocessing?

I have been told that logging can not be used in Multiprocessing. You have to do the concurrency control in case multiprocessing messes the log. But I did some test, it seems like there is no problem using logging in multiprocessing import…
Kramer Li
  • 2,284
  • 5
  • 27
  • 55
34
votes
5 answers

_multiprocessing.SemLock is not implemented when running on AWS Lambda

I have a short code that uses the multiprocessing package and works fine on my local machine. When I uploaded to AWS Lambda and run there, I got the following error (stacktrace trimmed): [Errno 38] Function not implemented: OSError Traceback (most…
Zach Moshe
  • 2,782
  • 4
  • 24
  • 40
33
votes
4 answers

Non-blocking multiprocessing.connection.Listener?

I use multiprocessing.connection.Listener for communication between processes, and it works as a charm for me. Now i would really love my mainloop to do something else between commands from client. Unfortunately listener.accept() blocks execution…
igann
  • 431
  • 4
  • 7
31
votes
5 answers

Multiprocessing python program inside Docker

I am trying to test multiprocessing for python inside a docker container but even, if the processes are created successfully (I have 8 CPUs and 8 processes are created), they always take only one physical CPU. Here is my code: from…
hanego
  • 1,595
  • 1
  • 14
  • 27
30
votes
2 answers

Dask: How would I parallelize my code with dask delayed?

This is my first venture into parallel processing and I have been looking into Dask but I am having trouble actually coding it. I have had a look at their examples and documentation and I think dask.delayed will work best. I attempted to wrap my…
Monty
  • 781
  • 2
  • 6
  • 23
28
votes
1 answer

PicklingError when using multiprocessing

I am having trouble when using the Pool.map_async() (and also Pool.map()) in the multiprocessing module. I have implemented a parallel-for-loop function that works fine as long as the function input to Pool.map_async is a "regular" function. When…
matiasq
  • 537
  • 1
  • 6
  • 12
27
votes
5 answers

Multiprocessing a function with several inputs

In Python the multiprocessing module can be used to run a function over a range of values in parallel. For example, this produces a list of the first 100000 evaluations of f. def f(i): return i * i def main(): import multiprocessing …
Mark Bell
  • 880
  • 1
  • 12
  • 22
27
votes
2 answers

How to create global lock/semaphore with multiprocessing.pool in Python?

I want limit resource access in children processes. For example - limit http downloads, disk io, etc.. How can I achieve it expanding this basic code? Please share some basic code examples. pool =…
Chameleon
  • 9,722
  • 16
  • 65
  • 127
26
votes
7 answers

How to efficiently run multiple Pytorch Processes / Models at once ? Traceback: The paging file is too small for this operation to complete

Background I have a very small network which I want to test with different random seeds. The network barely uses 1% of my GPUs compute power so i could in theory run 50 processes at once to try many different seeds at once. Problem Unfortunately i…
KoKlA
  • 898
  • 2
  • 11
  • 15
1
2
3
…
99 100