0

I'm trying to parallelize a task that involves a function which takes a custom object as a parameter. During the body of the function, said object gets manipulated, so I need a deepcopy of the original object at the start of each trial. When I use the code below, the markers that show if the object is changed don't show until ~ trial 30. I suppose I'm confused about how to properly transfer a copied object through functools.partial.

Host

object_base = Object()
object.initialize()
pool = multiprocessing.Pool(processes = 8)
return pool.map(functools.partial(processTrial, deepcopy(object_base),
                array1, array2), range(num_trials))

Worker

def processTrial(object, array1, array2, trial_number):
    if object.property != []:
        print '!!!!!!!!'
    for i in xrange(len(array1)):
        """ stuff including changing object.property """

As a side note, I also sometimes recieve the error "object instance as no attribute 'self' "... Any tips?

Philip Massey
  • 1,401
  • 3
  • 14
  • 24
  • What do you mean by "wearing off"? What exactly is supposed to be happening here? Most importantly, if you're trying to change `object.property` in `multiprocessing`, it's inherent to multiprocessing that changes in a child process may or may not be seen in other children or the parent; if you require _either_, your code is wrong. If you want to share data, you have to share it explicitly, as described in the `Sharing state` section. (I'm guessing that your `.map` call is picking a `chunksize` of ~30, and that's why you see different behavior around 30 than aroun d0, but that's not important.) – abarnert Aug 05 '13 at 19:29
  • 'object.property' is something that needs to change in order to calculate the return value of 'processTrial'. It changing should not effect another task of 'processTrial'. IE it might be '2' in one instance and '1' in the other, but must start as [] in each case. However, I think I found the workout for it, which is to have the deepcopy at the beginning of processTrial. – Philip Massey Aug 05 '13 at 19:59
  • If it must start as `[]` in each case, why not just set it to `[]` at the start of each case? Or, for that matter, why not create the `Object` itself in each case? – abarnert Aug 05 '13 at 20:10
  • Fair, but I'm looking to get as much speed out of this as possible. And merely setting the variable won't be enough, too many side effects also occur. Unless you can see something inherently wrong with executing a deepcopy at the beginning of processTrial instead of in pool.map, I'm just going to use that. – Philip Massey Aug 05 '13 at 20:13
  • 1
    It doesn't seem all that likely to me that the `Object` constructor actually takes a significant amount of time, but calling `deepcopy` on it, and then pickling it and unpickling it to send between processes, does not. But if that's the case, then OK, do the pickling and unpickling and deepcopying each time instead of the construction. – abarnert Aug 05 '13 at 20:32
  • It was the `object.initialization()` that did it. Thanks for your help though! – Philip Massey Aug 05 '13 at 20:37

0 Answers0