python object composition and multiprocessing

Question

If I implement object composition with getattr and pass the object into a new process, I get an RecursionError from getattr. Here is an example:

from multiprocessing import Pool

class Bar:
    def __init__(self, bar):
        self.bar = bar

class Foo:
    def __init__(self, bar):
        self._bar = bar

    def __getattr__(self, attr):
        try:
            return getattr(self._bar, attr)
        except RecursionError:
            print('RecursionError while trying to access {}'.format(attr))
            raise AttributeError

def f(foo):
    print(foo.bar)

if __name__ == '__main__':
    foo = Foo(Bar('baz'))
    f(foo)
    p = Pool(1)
    p.map(f, [foo])

The output is this:

baz
RecursionError while trying to access _bar
baz

Why doesn't Foo find the _bar attribute and has to resort to getattr?

What is this: `foo = Foo(Bar('baz'))`? What does it do in your opinion? — Elis Byberi, Nov 07 '17 at 18:09

score 3 · Answer 1 · answered Nov 07 '17 at 18:24

The problem is that Python needs to serialize the foo object and resurrect it in the new process. __init__ isn't called during resurrection, so your foo object won't have a ._bar attribute early enough.

The solution is to let the serialize/resurrect methods through specially, ie. change your __getattr__ to:

def __getattr__(self, attr):
    if attr in {'__getstate__', '__setstate__'}:
        return object.__getattr__(self, attr)
    return getattr(self._bar, attr)

and it should work.

python object composition and multiprocessing

1 Answers1