Stack Overflow
Stack Overflow

Questions tagged [multiprocessing]

Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. Relevant implementation and usage details vary per operating system and programming language. So always add tags for both the OS and language when using this tag.

Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. The term also refers to the ability of a system to support more than one processor and/or the ability to allocate tasks between them.

There are many variations on this basic theme, and the definition of multiprocessing can vary with context, mostly as a function of how CPUs are defined (multiple cores on one die, multiple dies in one package, multiple packages in one system unit, etc.).

Multiprocessing sometimes refers to the execution of multiple concurrent software processes in a system as opposed to a single process at any one instant. However, the terms multitasking or multiprogramming are more appropriate to describe this concept which is implemented mostly in software, whereas multiprocessing is more appropriate to describe the use of multiple hardware CPUs.

http://en.wikipedia.org/wiki/Multiprocessing

Multiprocessing may also refer to the Python multiprocessing module, which is a package that supports spawning processes using an API similar to Python's threading module. It offers both local and remote concurrency, effectively side-stepping the limitations of the Global Interpreter Lock by using sub-processes instead of threads.

14352 questions
5
votes
1 answer

Python, use multiprocessing to further speed up a cython function

the code shown here are simplied but triggers the same PicklingError. I know there is a lot discussion on what can and cannot be pickled, but I did find the solution from them. I write a simple cython script with the following function: def…
fast tooth
  • 2,317
  • 4
  • 25
  • 34
5
votes
3 answers

Thread Safety: Lock vs Reference

I have a C# program that has a list that does writes and reads in separate threads. The write is user initiated and can change the data at any random point in time. The read runs in a constant loop. It doesn't matter if the read is missing data in…
lolcodez
  • 659
  • 4
  • 9
  • 16
5
votes
4 answers

Python - Join Multiple Threads With Timeout

I have multiple Process threads running and I'd like to join all of them together with a timeout parameter. I understand that if no timeout were necessary, I'd be able to write: for thread in threads: thread.join() One solution I thought of was…
tonyduan
  • 75
  • 1
  • 3
5
votes
3 answers

Join a group of python processes with a timeout

I'm using python's multiprocessing library to create several processes. from multiprocessing import Process processes = [Process(target=function) for function in FUNCTIONS] for p in processes: p.start() I want them to run for some duration and…
Zags
  • 37,389
  • 14
  • 105
  • 140
5
votes
3 answers

Python Multiprocessing.Process how to reuse a process?

I'm using the python multiprocessing module to run some long running tasks in parallel. I'm using the start() method to run the job, but once the jobs have returned I'd like to run them again. Is it possible to reuse the Process I create? or do I…
Tim
  • 2,134
  • 3
  • 26
  • 40
5
votes
1 answer

EOFError with multiprocessing Manager

I have a bunch of clients connecting to a server via 0MQ. I have a Manager queue used for a pool of workers to communicate back to the main process on each client machine. On just one client machine having 250 worker processes, I see a bunch of…
Dustin Oprea
  • 9,673
  • 13
  • 65
  • 105
5
votes
2 answers

What happens when I multiprocessing.pool.apply_async more times than I have processors

I have the following setup: results = [f(args) for _ in range(10**3)] But, f(args) takes a long time to compute. So I'd like to throw multiprocessing at it. I would like to do so by doing: pool = mp.pool(mp.cpu_count() -1) # mp.cpu_count() ->…
inspectorG4dget
  • 110,290
  • 27
  • 149
  • 241
5
votes
2 answers

Why doesn't concurrent.futures make a copy of arguments?

My understanding was that concurrent.futures relied on pickling arguments to get them running in different processes (or threads). Shouldn't pickling create a copy of the argument? On Linux it does not seem to be doing so, i.e., I have to explicitly…
ariddell
  • 3,413
  • 34
  • 32
5
votes
2 answers

Concurrent i/o and processing for large data files in python

I am going to write a python program that reads chunks from a file, processes those chunks and then appends the processed data to a new file. It will need to read in chunks as the files to process will generally be larger than the amount of ram…
jramm
  • 6,415
  • 4
  • 34
  • 73
5
votes
2 answers

Python: How to make program wait till function's or method's completion

Often there is a need for the program to wait for a function to complete its work. Sometimes it is opposite: there is no need for a main program to wait. I've put a simple example. There are four buttons. Clicking each will call the same calculate()…
alphanumeric
  • 17,967
  • 64
  • 244
  • 392
5
votes
1 answer

Python multiprocessing.Pool map() "TypeError: string indices must be integers, not str"

I am attempting to use multiprocessing.Pool to do parallel processing on a list of dictionaries. An example is below (Please note: this is a toy example, my actual example will be doing cpu-intensive processing on the values in the actual…
Vector
  • 173
  • 3
  • 6
5
votes
1 answer

InternalError: current transaction is aborted, commands ignored until end of transaction block

I'm getting this error when doing database calls in a sub process using multiprocessing library. Visit : Pastie InternalError: current transaction is aborted, commands ignored until end of transaction block this is to a Postgre Database, using…
kevin
  • 4,177
  • 10
  • 32
  • 33
5
votes
3 answers

Python Multiprocessing And Argument Passing Help Needed

Hi I am trying to run the multiprocessing example in the docs: http://docs.python.org/3.4/library/concurrent.futures.html, the one using prime numbers but with a small difference. I want to be able to call a function with multiple arguments. What I…
davo36
  • 694
  • 9
  • 19
5
votes
1 answer

How to parallelize a for in python inside a class?

I have a python function funz that returns every time a different array of length p. I need to run this function different times and then to compute the mean of each value. I can do this with a for loop but it takes a lot of times. I am trying to…
Donbeo
  • 17,067
  • 37
  • 114
  • 188
5
votes
1 answer

multiprocessing.Queue deadlocks after "reader" process death

I've been playing with multiprocessing package and noticed that queue can be deadlocked for reading when: The "reader" process is using get with timeout > 0: self.queue.get(timeout=3) "reader" dies while get is blocking due to timeout. After that…
Michael
  • 481
  • 2
  • 6
  • 13
1 2 3
…
99
100