Python multiprocessing: Pool of custom Processes

Question

I am subclassing the Process class, into a class I call EdgeRenderer. I want to use multiprocessing.Pool, except instead of regular Processes, I want them to be instances of my EdgeRenderer. Possible? How?

Are you trying to write your code to use multi threading this way? — Brian Gianforcaro, Apr 11 '09 at 23:27

score 5 · Answer 1 · answered Dec 01 '16 at 18:50

This seems to work:

import multiprocessing as mp

ctx = mp.get_context()  # get the default context

class MyProcess(ctx.Process):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        print("Hi, I'm custom a process")

ctx.Process = MyProcess  # override the context's Process

def worker(x):
    print(x**2)

p = ctx.Pool(4)
nums = range(10)
p.map(worker, nums)

score 3 · Accepted Answer · answered Apr 12 '09 at 14:24

3

From Jesse Noller:

It is not currently supported in the API, but would not be a bad addition. I'll look at adding it to python2.7/2.6.3 3.1 this week

answered Apr 12 '09 at 14:24

Ram Rachum

84,019
84
236
374

score 2 · Answer 3 · answered Apr 12 '09 at 01:26

2

I don't see any hook for it in the API. You might be able to get away with replicating your desired functionality by using initializer and initargs argument. Alternately, you can build the functionality into the callable object that you use for mapping:

class EdgeRenderTask(object):
    def op1(self,*args):
        ...
    def op2(self,*args):
        ...
p = Pool(processes = 10)
e = EdgeRenderTask()
p.apply_async(e.op1,arg_list)
p.map(e.op2,arg_list)

answered Apr 12 '09 at 01:26

David Berger

12,385
6
38
51

You can't pass a class as an argument to `p.map`, you'll get pickle error. The only way I've found around this is to implement a sort of custom `pool.map`....setting up and feeding the processes myself. Unfortunately it's gonna take more lines. – catwalker333 Jan 12 '14 at 20:16

Python multiprocessing: Pool of custom Processes

3 Answers3