Python Multiprocessing custom manager with associated objects

Question

I'm trying to make a class object usable for multiple processes. Unfortunarely this seems to be more of an issue than I anticipated.

I have the following class object:

class BusObject:

    inputs: IOObject
    outputs: IOObject

    def __init__(self):
        self.inputs = IOObject()
        self.outputs = IOObject()

with the associated object IOObject

class IOObject:

    idx: List[int]              # signal index
    tag: List[str]              # signal tag

    def __init__(self):

        self.idx = []
        self.tag = []

This combination worked fine. But now I run into the requirement that I have to make the BusObject available to multiple processes.

Therefore I created a custom multiprocessing.manager

class CustomManager(BaseManager):
    pass

def main():

    manager = CustomManager()

    # Registration of custom classes to manager
    manager.register('BusObject', BusObject)
    # manager.register('IOObject', IOObject)

    manager.start()
    
    busObject = manager.BusObject()

Works - almost ...

The problem is that the associated objects don't seem to be registered as well.

I tried to register them, too, but even if I do I run into the error

AttributeError
'AutoProxy[BusObject]' object has no attribute 'inputs'

Any ideas?

Then a nested class? I'm not sure of the correct terminology here. And would that make a difference in this particular case? — Viktor Katzy, Oct 14 '22 at 11:34
These managed objects are a bit though to deal with. One has to think if your system design _really_ needs then. I have some years on Python programming and consulting, and I would not advise trying to manipulate instances, methods and attributes of a shared object like this, and if so, it would take a _very_ careful design. — jsbueno, Oct 14 '22 at 14:16
The docs are short of examples and one has to do some guessing - but by registering an object to a manager, you are implying the creation of a proxy class - this proxy class has to "know" which attributes should also be shared - and managed objects. — jsbueno, Oct 14 '22 at 14:19

jsbueno · Accepted Answer · 2022-10-14T15:15:15.767

Upon reading the docs here - https://docs.python.org/3/library/multiprocessing.html#multiprocessing.managers.BaseManager.register - I 'd say you won't be able to access the .inputs and .outputs attributes directly - you can code your object like that, and the "owner" process of the real BusObject instance can do so - but in order to get instances for the IOObjects that are associated to a BUSObject (by the way, the term you are missing is this: associated objects, not "nested subclasses") - you have to retrieve then through methods that are registered in the method_to_typeid parameter of the call to .register().

So, you'd do

    ...
    def get_inputs(self):
        return self.inputs
    # same for outputs...

method (which otherwise looks silly in Python), and pass {'get_inputs': 'IOObject', 'get_outputs': 'IOObject'} as the method_to_typeid argument on the call to register.

That way, the other defaults of registering a class will create a proxy for BUSObject that will be able to return proxies to IOObject when you call these methods (but busobj_instance.inputs should remain opaque).

It is even possible this can play along with @property so you can register the property getter method as a proxy-returner and be able to use the names as attributes.

Of course, the IOOBject class must be properly registered as well.

Sorry for not providing a working example and testing out the possibilities - but that would take me at least a couple hours to get going.

Thank you very much, just tested your suggestions and it worked immediately. Also added a setter method for both assosiated objects and corrected the terminology in my question accordingly — Viktor Katzy, Oct 17 '22 at 05:03

score 0 · Answer 2 · answered Oct 14 '22 at 13:39

0

Since BusObject contains IOObject, try registering IOObject first, then BusObject.

answered Oct 14 '22 at 13:39

Roland Smith

42,427
3
64
94

Thank you! Unfortunately it wasn't as simple as that - tried it first before implementing the suggestion by jsbueno – Viktor Katzy Oct 17 '22 at 05:04

Python Multiprocessing custom manager with associated objects

2 Answers2