Mysterious pickle error while profiling a multi-process Python script

Question

I"m using the multiprocessing module, and I'm using UpdateMessage objects (my own class), sent through multiprocessing.Queue objects, to communicate between processes. Here's the class:

class UpdateMessage:
    def __init__(self, arrayref, rowslice, colslice, newval):
        self.arrayref = arrayref
        self.rowslice = rowslice
        self.colslice = colslice
        self.newval = newval
    def do_update(self):
        if self.arrayref == 'uL':
            arr = uL
        elif self.arrayref == 'uR':
            arr = uR
        else:
            raise Exception('UpdateMessage.arrayref neither uL nor uR')
        arr[self.rowslice, self.colslice] = self.newval

When I run the script, it works perfectly fine. However, when I run it with either cProfile or profile, it gives the following error:

_pickle.PicklingError: Can't pickle <class '__main__.UpdateMessage'>: attribute lookup __main__.UpdateMessage failed

It seems to be trying to pickle the class, but I can't see why this is happening. My code doesn't do this, and it works just fine without it, so it's probably the multiprocessing module. But why would it need to pickle UpdateMessage, and how can I fix the error?

EDIT: here's part of the code that's sending the UpdateMessage (multiple parts of the script do this, but all in the same way):

msg = UpdateMessage(uLref, refer[0] + marker[0] - 2,
                    slice(uL.shape[1]), ustar.copy())
queue.put(msg)

The traceback isn't very helpful:

Traceback (most recent call last):
  File "/usr/lib/python3.2/multiprocessing/queues.py", line 272, in _feed
    send(obj)

The reason `UpdateMessage` is getting pickled is that the interprocess communication path is just a byte stream, so class instances need to be serialized to be sent to another process. I don't know why `profile` interferes with that, though. — Russell Borogove, Jul 16 '12 at 22:51
I have got a similar problem: http://stackoverflow.com/q/41892297/1878788 This could be related to the following bug: http://bugs.python.org/issue9914 — bli, Jan 30 '17 at 12:21

score 5 · Answer 1 · answered Jul 16 '12 at 22:32

I do not know how your processes look like but:

'__main__.UpdateMessage'

refers to UpdateMessage in the launched module.

When another process is started not with the Module of the class UpdateMessage, UpdateMessage will not be available.

you have to import that module that includes UpdateMessage so UpdateMessage.__module__ is not '__main__'.

Then Pickle can find UpdateMessage also in other programs.

I recommend the __main__.py to look like this:

import UpdateMessage_module

UpdateMessage_module.main()

score 4 · Answer 2 · answered May 02 '14 at 08:16

4

I had the same problem, and solved it by defining the class to be pickled in its own file.

answered May 02 '14 at 08:16

mpenkov

21,621
10
84
126

1

Your advice worked for me for functions that were attributes of a class that had to be pickled. I had errors like `_pickle.PicklingError: Can't pickle : attribute lookup count_annot on __main__ failed` – bli Jan 30 '17 at 12:59

score 0 · Answer 3 · answered Jul 16 '12 at 21:43

0

I'm assuming the pickling attempt on your class is happening before your class is sent to another process. The easiest solution is probably just to implement the pickle-protocol explicitly on your class...

answered Jul 16 '12 at 21:43

thebjorn

26,297
11
96
138

Mysterious pickle error while profiling a multi-process Python script

3 Answers3

Linked