Accessing full dictionary when setting a nested dictionary item in Python

Question

I'm trying to create a dictionary subclass which allows a dictionary A to be created which will update a value in a pre-exiting dictionary B to equal a string representation of dictionary A. I see it as an observer pattern, without the ability to have multiple objects observing.

i.e.:

import json
from collections import Mapping


class ObservedDict(dict):

    def __init__(self, initial_dict, name=None, observer=None, top_level=True):
        for k, v in initial_dict.items():
            if isinstance(v, dict):
                initial_dict[k] = ObservedDict(v, name, observer, top_level=False)

        super().__init__(initial_dict)

        self.name = name
        self.observer = observer
        if top_level is True:  # initialise the key:value pair in B
            observer[name] = json.dumps(initial_dict)

    def __setitem__(self, item, value):
        if isinstance(value, dict):
            _value = ObservedDict(value, self.name, self.observer, top_level=False)
        else:
            _value = value

        super().__setitem__(item, _value)
        # Update B
        self.observer[self.name] = json.dumps(self)

B = {}
A = ObservedDict({'foo': 1, 'bar': {'foobar': 2}}, 'observed', B)

B is now {'observed': '{"foo": 1, "bar": {"foobar": 2}}'}and A is {'foo': 1, 'bar': {'foobar': 2}}. There are three cases for updating a value in the dictionary (ignoring update and set for now):

I can update A's top-level keys, and it works just fine:

A['foo'] = 2
# B is now automatically {'observed': '{"foo": 2, "bar": {"foobar": 2}}'}

I can update the entirety of a nested dictionary:

A['bar'] = {'foobar': 4}
# B is now automatically {'observed': '{"foo": 2, "bar": {"foobar": 4}}'}

But, if I edit a nested value by using the [] method, self in __setitem__ is the nested dict, not the whole dictionary with which the ObservedDict class is initialised, so:

A['bar']['foobar'] = 4
# B is now {'observed': '{"foobar": 4}'}

My question is: how do I retain information about the parent dictionary (i.e. the one used to initialise the class) such that on setting a value using the third case, dictionary B will update and include the whole of dictionary A (matching case 2, in this instance)?

When creating the nested `ObservedDict` it needs an additional parameter to tell it its parent dictionary. The nested dict should then only inform its parent about a change so that the parent (or maybe an ancestor further upwards} handles the change correctly. — Michael Butscher, Jan 27 '19 at 22:06
@jonrsharpe It is an `ObservedDict` (i.e. my subclass), but yes, it has no connection with `A`, hence why I can't access A from `A['bar']`. But that's my question: how do I connect `A['bar']` (and any other generic nested dictionary) with `A`? — Bryn Pickering, Jan 27 '19 at 22:10

score 1 · Accepted Answer · answered Jan 27 '19 at 23:15

One thing you can do to make the class simpler is externalize the behavior of updating B, like so:

class ObservedDict(dict):
    def __init__(self, initial_dict, on_changed=None):
        super().__init__(initial_dict)

        self.on_changed = on_changed

        for k, v in initial_dict.items():
            if isinstance(v, dict):
                super().__setitem__(
                    k, ObservedDict(v, on_changed=self.notify))

        self.notify()

    def __setitem__(self, key, value):
        if isinstance(value, dict):
            value = ObservedDict(value, on_changed=self.notify)
        super().__setitem__(key, value)
        self.notify()

    def notify(self, updated=None):
        if self.on_changed is not None:
            self.on_changed(self)

Then you can use it with a lambda:

import json


B = {}
A = ObservedDict(
        {'foo': 1, 'bar': {'foobar': 2}},
        lambda d: B.update({'observed': json.dumps(d)}))

print(B)
A['foo'] = 2
print(B)
A['bar'] = {'foobar': 4}
print(B)
A['bar']['foobar'] = 5
print(B)

Or with a child class

class UpdateObserverDict(ObservedDict):
    def __init__(self, *args, name, observer, **kwargs):
        self.observer = observer
        self.name = name
        super().__init__(*args, **kwargs)

    def notify(self, updated=None):
        self.observer[self.name] = json.dumps(self)

B = {}
A = UpdateObserverDict(
        {'foo': 1, 'bar': {'foobar': 2}},
        name='observed', observer=B)

print(B)
A['foo'] = 2
print(B)
A['bar'] = {'foobar': 4}
print(B)
A['bar']['foobar'] = 5
print(B)

both of which give you the expected result:

{'observed': '{"foo": 1, "bar": {"foobar": 2}}'}
{'observed': '{"foo": 2, "bar": {"foobar": 2}}'}
{'observed': '{"foo": 2, "bar": {"foobar": 4}}'}
{'observed': '{"foo": 2, "bar": {"foobar": 5}}'}

Thanks! I quite like the child class option. I can't get my head around how `notify` works, though. in the case of a nested dict, it is calling `self.notify(self.notify)`. How does that filter up to, e.g. the lambda function? Also, I can see it raises an exception if I remove `updated=None`, but I can't see what the additional argument is in there... — Bryn Pickering, Jan 29 '19 at 08:48
In a nested dict it is essentially calling `self.notify(parent.notify)`, since each dict acting as a parent parent passes its own `self.notify` (bound to itself) to the `ObservedDict`s it creates. Any call propagates up until the top-most which either received `on_changed` from outside or is actually an instance of your child class which overrides `notify`. — Chris Hunt, Jan 30 '19 at 02:33
The `updated` argument is required because we want the lambda to have the instance as an argument. We want this because if we didn't pass it as an argument we would not be able to reference the instance itself from the inline lambda as in example 1 (`A` is not yet defined when the lambda is invoked the first time). We give it a default argument because we want to be able to call `self.notify()` without passing `self` - since passing `self` to ones own method would look a little silly. You can get rid of it if you change `self.on_changed(self)` to `self.on_changed()`. — Chris Hunt, Jan 30 '19 at 02:36

Bryn Pickering · Answer 2 · 2019-01-27T23:07:37.927

OK, so although I had played around with attaching the parent dictionary to the nested dictionaries before, without luck, @MichaelButscher's comment spurred me to try again. Below is a working solution, which seems to work for setting a value in a nested dictionary using the [] method, no matter the depth.

import json
from collections import Mapping


class ObservedDict(dict):

    def __init__(self, initial_dict, name=None, observer=None, parent=None):
        for k, v in initial_dict.items():
            if isinstance(v, dict):
                _parent = self if parent is None else parent
                initial_dict[k] = ObservedDict(v, name, observer, parent=_parent)

        super().__init__(initial_dict)

        self.observer = observer
        self.name = name
        self.parent = parent
        if parent is None:  # initialise the key:value pair in B
            observer[name] = json.dumps(initial_dict)

    def __setitem__(self, item, value):
        if isinstance(value, dict):
            _value = ObservedDict(value, self.name, self.observer, parent=self.parent)
        else:
            _value = value

        super().__setitem__(item, _value)

        # Update B
        if self.parent is not None:
            self.observer[self.name] = json.dumps(self.parent)  # nested dict
        else:
            self.observer[self.name] = json.dumps(self)  # the top-level dict

Ensuring the 'parent' was always the self as given when initialising the object for the first time (i.e. A) seems to do the trick.

Accessing full dictionary when setting a nested dictionary item in Python

2 Answers2