Stack Overflow
Stack Overflow

Questions tagged [python-multiprocessing]

multiprocessing is a package that supports spawning processes using an API similar to the threading module in python programming language. When asking a question about this topic, please mention the operating system you are running it on, or add it to the tags.

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using Operating System processes instead of threads.

The multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

4758 questions
26
votes
2 answers

how to to terminate process using python's multiprocessing

I have some code that needs to run against several other systems that may hang or have problems not under my control. I would like to use python's multiprocessing to spawn child processes to run independent of the main program and then when they…
Dan Littlejohn
  • 1,329
  • 4
  • 16
  • 30
26
votes
2 answers

Python: TypeError: Pickling an AuthenticationString object is disallowed for security reasons

I'm creating an object of a class(with multiprocessing) and adding it to a Manager.dict() so that I can delete the item from the dictionary inside the object (the item points to) when its work completes.. I tried the following code: from…
RatDon
  • 3,403
  • 8
  • 43
  • 85
26
votes
3 answers

Preserve custom attributes when pickling subclass of numpy array

I've created a subclass of numpy ndarray following the numpy documentation. In particular, I have added a custom attribute by modifying the code provided. I'm manipulating instances of this class within a parallel loop, using Python…
Gabriel
  • 1,870
  • 1
  • 19
  • 20
25
votes
3 answers

How to handle SQLAlchemy Connections in ProcessPool?

I have a reactor that fetches messages from a RabbitMQ broker and triggers worker methods to process these messages in a process pool, something like this: This is implemented using python asyncio, loop.run_in_executor() and…
roman
  • 453
  • 6
  • 11
24
votes
2 answers

Where is documentation for multiprocessing.pool.ApplyResult?

There is frighteningly little strict API documentation (read: ZERO) for multiprocessing.pool.ApplyResult. The multiprocessing explanation doc talks about ApplyResults, but does not define them. The same appears to apply to multiprocessing.pool.Pool,…
Mark Gerolimatos
  • 2,424
  • 1
  • 23
  • 33
24
votes
4 answers

celery: daemonic processes are not allowed to have children

In Python (2.7) I try to create processes (with multiprocessing) in a celery task (celery 3.1.17) but it gives the error: daemonic processes are not allowed to have children Googling it, I found that most recent versions of billiard fix the "bug"…
Patrick
  • 2,577
  • 6
  • 30
  • 53
24
votes
2 answers

How can I abort a task in a multiprocessing.Pool after a timeout?

I am trying to use the multiprocessing package of python in this way: featureClass = [[1000, k, 1] for k in drange(start, end, step)] #list of arguments for f in featureClass: pool.apply_async(worker, args=f,…
farhawa
  • 10,120
  • 16
  • 49
  • 91
23
votes
2 answers

Why is `multiprocessing.Queue.get` so slow?

I need help in understanding multiprocessing.Queue. The problem I'm facing is that getting results from queue.get(...) are hilariously behind compared to calls to queue.put(...) and the queue's buffer (the deque). This leaking abstraction led me to…
JBSnorro
  • 6,048
  • 3
  • 41
  • 62
23
votes
2 answers

Leveraging "Copy-on-Write" to Copy Data to Multiprocessing.Pool() Worker Processes

I have a bit of multiprocessing Python code that looks a bit like this: import time from multiprocessing import Pool import numpy as np class MyClass(object): def __init__(self): self.myAttribute = np.zeros(100000000) # basically a big…
alexrson
  • 233
  • 2
  • 5
23
votes
5 answers

obtaining pid of child process

I am using python's multiprocessing module to spawn new process as follows : import multiprocessing import os d = multiprocessing.Process(target=os.system,args=('iostat 2 > a.txt',)) d.start() I want to obtain pid of iostat command or the command…
tazim
  • 585
  • 2
  • 7
  • 17
23
votes
3 answers

Does multiprocessing.pool.imap has a variant (like starmap) that allows for multiple arguments?

I am doing some calculations on large collections of bytes. The process runs on chunks of bytes. I am trying to use parallel processing using multiprocessing for performance enhancement. Initially I tried to use pool.map but that only allows single…
Abdul Qadir
  • 439
  • 1
  • 6
  • 12
23
votes
2 answers

Python multiprocessing.cpu_count() returns '1' on 4-core Nvidia Jetson TK1

Can anyone tell me why Python's multiprocessing.cpu_count() function would return 1 when called on a Jetson TK1 with four ARMv7 processors? >>> import multiprocessing >>> multiprocessing.cpu_count() 1 The Jetson TK1 board is more or less straight…
Hephaestus
  • 1,982
  • 3
  • 27
  • 35
23
votes
4 answers

Can I use map / imap / imap_unordered with functions with no arguments?

Sometimes I need to use multiprocessing with functions with no arguments. I wish I could do something like: from multiprocessing import Pool def f(): # no argument return 1 # TypeError: f() takes no arguments (1 given) print Pool(2).map(f,…
usual me
  • 8,338
  • 10
  • 52
  • 95
21
votes
3 answers

Can I run multiprocessing Python programs on a single core machine?

So this is more or less a theoretical question. I have a single core machine which is supposedly powerful but nevertheless only one core. Now I have two choices to make : Multithreading: As far as my knowledge is concerned I cannot make use of…
Subhayan Bhattacharya
  • 5,407
  • 7
  • 42
  • 60
21
votes
10 answers

Python Shared Memory Dictionary for Mapping Big Data

I've been having a hard time using a large dictionary (~86GB, 1.75 billion keys) to process a big dataset (2TB) using multiprocessing in Python. Context: a dictionary mapping strings to strings is loaded from pickled files into memory. Once loaded,…
Jon Deaton
  • 3,943
  • 6
  • 28
  • 41
1 2
3
…
99 100