1

I have a multiprocessing namespace initialized as the following

from multiprocessing import Manager
manager = Manager()
manager_namespace = manager.Namespace()
manager_namespace.__setattr__('x', 1)
manager_namespace.__setattr__('y', 2)
manager_namespace.__setattr__('z', 3)

I can access these attributes just fine individually by using

manager_namespace.x

but I would like a way to be able to iterate through all set attributes so I can modify them in batches without having to type manager_namespace.(x/y/z) every time. I used VSCode to try and find out where these attributes are stored in the namespace object but couldn't find the location. If anyone has any insights, they would be much appreciated!

Lafonz
  • 23
  • 6

1 Answers1

2

As you have probably figured out, the problem is that manager_namespace is actually a proxy object and inspecting its __dict__ attribute will not reveal any useful information. Therefore, there is no general solution to your problem that can work for an arbitrary Namespace proxy.

However, if you look at the documentation for multiprocessing.managers.Namespace, you will see (in part):

A namespace object has no public methods, but does have writable attributes. Its representation shows the values of its attributes.

However, when using a proxy for a namespace object, an attribute beginning with '_' will be an attribute of the proxy and not an attribute of the referent:

So, as long as:

  1. Your attributes do not begin with an underscore ('_') but otherwise are legal variable names and
  2. The values of the attributes are not values that contain an equal sign ('=') then

You should be able to take the string representation of the Namespace proxy and with a simple regex pull out all the attribute names:

from multiprocessing import Manager
import re

# required for Windows:
if __name__ == '__main__':
    manager = Manager()
    manager_namespace = manager.Namespace()
    manager_namespace.__setattr__('x', 1)
    manager_namespace.__setattr__('y', 2)
    manager_namespace.__setattr__('z', 3)

    attributes = re.findall(r'\b([A-Za-z][A-Za-z_0-9]*)=', str(manager_namespace))
    print(attributes)

Prints:

['x', 'y', 'z']

You could relax the condition for naming your attributes but would have to then adjust the regular expression accordingly.

So it is only under this extremely limited and possibly not very useful set of circumstances, assuming that you have control over them, that getting the attributes could be feasible.

Update

Another possibility is to use a new, customized managed class MyNamespace (call it whatever you want) in place of Namespace that supports a get_dict (call it whatever you want) method that will return the dictionary of the actual namespace object:

from multiprocessing.managers import BaseManager, NamespaceProxy

class MyNamespaceManager(BaseManager):
    pass

class MyNamespace:
    def get_dict(self):
        return self.__dict__

class MyNamespaceProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'get_dict')

    def get_dict(self):
        return self._callmethod('get_dict')

# required for Windows:
if __name__ == '__main__':
    MyNamespaceManager.register('MyNamespace', MyNamespace, MyNamespaceProxy)
    with MyNamespaceManager() as manager:
        ns = manager.MyNamespace()
        ns.__setattr__('x', 1)
        ns.__setattr__('y', 2)
        ns.z = 3 # Different style of assignment

        print(ns.get_dict())

Prints:

{'x': 1, 'y': 2, 'z': 3}

If you will be using the SyncManager class because you need other built-in managed objects, for example, dict, then:

from multiprocessing import Manager
from multiprocessing.managers import SyncManager, NamespaceProxy

class MyNamespace:

    def get_dict(self):
        return self.__dict__

class MyNamespaceProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'get_dict')

    def get_dict(self):
        return self._callmethod('get_dict')

# required for Windows:
if __name__ == '__main__':
    SyncManager.register('MyNamespace', MyNamespace, MyNamespaceProxy)
    with Manager() as manager:
        ns = manager.MyNamespace()
        ns.__setattr__('x', 1)
        ns.__setattr__('y', 2)
        ns.z = 3 # Different style of assignment

        print(ns.get_dict())
Booboo
  • 38,656
  • 3
  • 37
  • 60
  • Thank you for the help! I did notice that this is a proxy object and therefore its __dict__ has no useful information. I'm not very comfortable with regex so I didn't think to use that. I want to mention I found a hacky way of doing it by copying the proxy to a new variable via the deepcopy module. After doing this I could then iterate through the __dict__ of the copied variable. – Lafonz Feb 04 '22 at 20:52
  • I don't blame you; as I said, I didn't think it was of much practical use. Also, assuming you do know what the attributes are, I wouldn't use your "hacky" way either. – Booboo Feb 04 '22 at 21:07
  • Apart from it not being very time/space efficient, why wouldn't you use my "hacky" method? Thanks again for the help. – Lafonz Feb 04 '22 at 21:20
  • You said it: It's not efficient. – Booboo Feb 04 '22 at 21:22
  • Gotcha, I might try implementing it via your method then. I do 'know' all of the attributes, there are just a lot of them so this would make it much easier and reduce lines of code needed. – Lafonz Feb 04 '22 at 21:25
  • I have added an update, which will be of interest. Whether it is of practical value is another question. – Booboo Feb 04 '22 at 21:48