Stack Overflow
Stack Overflow

Questions tagged [multiprocessing]

Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. Relevant implementation and usage details vary per operating system and programming language. So always add tags for both the OS and language when using this tag.

Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. The term also refers to the ability of a system to support more than one processor and/or the ability to allocate tasks between them.

There are many variations on this basic theme, and the definition of multiprocessing can vary with context, mostly as a function of how CPUs are defined (multiple cores on one die, multiple dies in one package, multiple packages in one system unit, etc.).

Multiprocessing sometimes refers to the execution of multiple concurrent software processes in a system as opposed to a single process at any one instant. However, the terms multitasking or multiprogramming are more appropriate to describe this concept which is implemented mostly in software, whereas multiprocessing is more appropriate to describe the use of multiple hardware CPUs.

http://en.wikipedia.org/wiki/Multiprocessing

Multiprocessing may also refer to the Python multiprocessing module, which is a package that supports spawning processes using an API similar to Python's threading module. It offers both local and remote concurrency, effectively side-stepping the limitations of the Global Interpreter Lock by using sub-processes instead of threads.

14352 questions
5
votes
1 answer

Python Logging with Multiprocessing in Windows

I have a rather large Python project which currently runs on Linux but I am trying to expand to Windows. I've reduced the code to a full example which can be run to illustrate my problems: I have two classes, Parent and Child. Parent is initialized…
user2093082
  • 407
  • 8
  • 19
5
votes
1 answer

Python class inheriting multiprocessing: mocking one of the class objects

I have written a class that inherits the multiprocessing.Process() class. In the initialization I set some parameters, one of them is another class that writes to some file on my hard drive. For the purpose of unit testing I would like to mock this…
Chris
  • 899
  • 2
  • 17
  • 25
5
votes
2 answers

What's a Pythonic way to make a non-blocking version of an object?

I often use python objects with methods that block until finished, and want to convert these methods to non-blocking versions. I find myself executing the following pattern quite frequently: Define the object Define a function which creates an…
Andrew
  • 2,842
  • 5
  • 31
  • 49
5
votes
1 answer

python multiprocessing queue error

I have this python code to read a file, do some processing and write the results in parallel: def line_chunker(path): """ Reads a file in chunks and yields each chunk. Each chunk is guaranteed to end at a carriage return (EOL). Each…
jramm
  • 6,415
  • 4
  • 34
  • 73
5
votes
0 answers

python-igraph multiprocessing task

I'm using python-igraph for my task. Let's say, we have a big weighted directed graph and for each node we want to mark out it's edges with a special flag (add an attribute to it) if it's weight is significant (for example top-n by weight).…
Efimovgk
  • 61
  • 3
5
votes
2 answers

How to work around the Queue corruption when using Process.Terminate()

I'm building a Python script/application which launches multiple so-called Fetchers. They in turn do something and return data into a queue. I want to make sure the Fetchers don't run for more than 60 seconds (because the entire application runs…
Thijs Cramer
  • 87
  • 1
  • 8
5
votes
1 answer

Python Multiprocessing Early Termination

While my script is running, it is possible that an error may occur at some point. In that case, all processes should be properly terminated, an error message should be returned, and the script should exit. The code I have now seems to not fulfill…
fire_water
  • 1,380
  • 1
  • 19
  • 33
5
votes
2 answers

Python 3.4 multiprocessing recursive Pool.map()

I'm developing with Python 3.4 on Ubuntu 14.04. I was trying to do recursive Pool.map(). After I invoke g(), it hangs there and never returns. import multiprocessing as mp pool = mp.Pool() def d(x): return x / 2.0 def f(x): w =…
Jen
  • 163
  • 1
  • 8
5
votes
1 answer

Pickling error: Can't pickle

I am wondering what this error might mean: PicklingError: Can't pickle : attribute lookup __builtin__.function failed I understand that it has something to do with using multiple cores. I am running my program on a cluster and…
naomig
  • 271
  • 1
  • 7
  • 12
5
votes
1 answer

Why does this implementation of multiprocessing.pool not work?

Here is the code I am using: def initFunction(arg1, arg2): def funct(value): return arg1 * arg2 * value return funct os.system("taskset -p 0xff %d" % os.getpid()) pool = Pool(processes=4) t = np.linspace(0,1,10e3) a,b,c,d,e,f,g,h…
bynx
  • 760
  • 2
  • 8
  • 19
5
votes
1 answer

'Collection' object is not callable error in pymongo with process Pool

Using the code below causes: 'Collection' object is not callable. If you meant to call the '__getnewargs__' method on a 'Collection' object it is failing because no such method exists. The code : from multiprocessing import Pool db =…
rok
  • 9,403
  • 17
  • 70
  • 126
5
votes
1 answer

Ways to free memory back to OS from Python?

I have code that looks similar to this: def memoryIntensiveFunction(x): largeTempVariable = Intermediate(x) processFunction(largeTempVariable,x) The problem is that the variable temp is something like 500 mb in a test case of mine, but that…
amos
  • 5,092
  • 4
  • 34
  • 43
5
votes
1 answer

Scons AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'

I have an sconstruct script that is instantiating an object.This object internally calls a method than run multiprocessing module. example shown below This object before calling the function unpickles a file and pass the inputs to multiprocessing…
Sri
  • 693
  • 1
  • 9
  • 22
5
votes
1 answer

Python ZMQ and multiprocessing causes zmq.error.ZMQError: Interrupted system call

I have a Python script where I'm binding several (e.g., 5) ZMQ receiver sockets like so: receiver_1 = context.socket(zmq.PULL) receiver_1.bind("tcp://*:5555") ... receiver_5 = context.socket(zmq.PULL) receiver_5.bind("tcp://*:5559") receivers =…
Marvin
  • 51
  • 4
5
votes
3 answers

What does "pcntl_fork(): Error 12" mean?

I've searched until I was blue in the face and cannot find the answer to this question. Where I can find a table listing the meanings of all the error codes for pcntl_fork()? Or even the C fork() function, for that matter.
Jonathon Hill
  • 3,445
  • 1
  • 33
  • 31
1 2 3
…
99
100