How to access property attribute from multiprocess Proxy class in Python?

Question

I am trying to get attribute from my Proxy class but I don't quite understand the implementation (as per https://docs.python.org/3.9/library/multiprocessing.html?highlight=multiprocessing#multiprocessing.managers.BaseManager.register)

I understand I need to pass exposed and method_to_typeid to the .register() method because if I don't I only have access to "public" methods and not attributes.

Here is my code;

from multiprocessing import Process
from multiprocessing.managers import BaseManager

class CustomManager(BaseManager):
    # nothing
    pass

class TestClass:

    def __init__(self):
       self._items = []

    @property
    def items(self):
        return self._items

    def fill_items(self):
        self._items.append(1)


if __name__ == "__main__":
    CustomManager.register(
        'TestClass', 
        TestClass,
        exposed=('items', 'fill_items'),
        method_to_typeid={'items': 'list'}
    )
    manager = CustomManager()
    manager.start()
    shared_object = manager.TestClass()
    
    p = Process(target=shared_object.fill_items)
    p.start()
    p.join()
    print(shared_object.items)
    #print(shared_object.items())

I would expect this to return my list but it returns a reference to the method;
Output:

<bound method items of <AutoProxy[TestClass] object, typeid 'TestClass' at 0x7feb38056670>>

But when I try to call it as a method i.e. shared_object.items() I get;

File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/managers.py", line 824, in _callmethod
    raise convert_to_error(kind, result)
TypeError: 'list' object is not callable

Which makes sense because it is an attribute that contains list type value not a method. So but then why when I try to call it as an attribute I get it's reference not the value?

Tried following the official documentation and checked already asked questions, for most of which the solution was to add NamespaceProxy, but it looks like now instead of implementing our own NamespaceProxy the correct way is to just pass the two extra args for .register() method.

score 0 · Accepted Answer · answered Jan 27 '23 at 14:57

0

The solution here is just to use threading instead of multiprocessing. ChatGPT got pretty close to the implementation I needed but could not resolve the issue without changing the implementation of my classes. In the end it makes more sense to use threads anyway because;

threads share memory as opposed to processes
my script is I/O bound which suggests using threads where as CPU bound ones should use processes

answered Jan 27 '23 at 14:57

kristapsm1111

11
3

but "threading" is fundamentally differnet from "multiprocessing" and one can't always just replace one for the other. It is better know what one is a doing instead of relaying on the output of a tool which generates code you can't understand. A property won't work out-of-the box with a manager - a first approach is to use a plain "getter " method instead of a property: should work without surprises. – jsbueno Jan 27 '23 at 15:02

How to access property attribute from multiprocess Proxy class in Python?

1 Answers1