1

I have inherited a bunch of legacy code and have run into a bit of a breaking issue. I believe the idea attempted here was to use a containerized cache such as redis to hold certain attributes to be shared across many processes. Apart from whether or not this was the best way to accomplish the task, I stumbled upon something that confused me. If the __getattribute__ and __setattr__ methods are overridden as below, mutable instance attributes are not able to be updated. I would expect the changes made to self.b to be reflected in the print() call but as you can see from the output, only the initial blank dictionary is printed. Stepping through the self.b["foo"] = "bar" line shows that a call to __getattribute__ is made and the correct object is retrieved from the cache but no call to __setattr__ is made nor does the dictionary seem to update in any way. Anyone have any ideas why this may be?

import dill

_reserved_fields = ["this_is_reserved"]

class Cache:
    _cache = {}

    def get(self, key):
        try:
            return self._cache[key]
        except KeyError:
            return None

    def set(self, key, value):
        self._cache[key] = value

    def exists(self, key):
        return key in self._cache


class Class(object):
    def __init__(self, cache):
        self.__dict__['_cache'] = cache
        self.a = 1
        self.a = 2
        print(self.a)
        self.b = dict()
        self.b["foo"] = "bar"  # these changes aren't set in the cache and seem to disappear
        print(self.b)
        self.this_is_reserved = dict()
        self.this_is_reserved["reserved_field"] = "value"  # this updates fine
        print(self.this_is_reserved)

    def __getattribute__(self, item):
        try:
            return object.__getattribute__(self, item)
        except AttributeError:
            key = "redis-key:" + item
            if not object.__getattribute__(self, "_cache").exists(key):
                raise AttributeError
            else:
                obj = object.__getattribute__(self, "_cache").get(key)
                return dill.loads(obj)

    def __setattr__(self, key, value):
        if key in _reserved_fields:
            self.__dict__[key] = value
        elif key.startswith('__') and key.endswith('__'):
            super().__setattr__(key, value)
        else:
            value = dill.dumps(value)
            key = "redis-key:" + key
            self._cache.set(key, value)

if __name__ == "__main__":
    cache = Cache()
    inst = Class(cache)

    # output
    # 2
    # {}
    # {'reserved_field': 'value'}
ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
Kyle
  • 11
  • 1

2 Answers2

0

The line self.b["foo"] = "bar" is getting the attribute b from self, where b is assumed to be a dictionary, and assigning to key "foo" in said dictionary. __setattr__ will only be invoked if you assign directly to b, e.g., self.b = {**self.b, 'foo': 'bar'} (in Python 3.5+ for the dictionary update).

Without adding hooks in your stored objects to trigger a cache save on their __setattr__, and their sub-objects __settattr__, etc..., you could add a way to explicitly trigger a cache update (meaning user code needs to be aware of the cache), or avoid mutating the fields except through direct assignment.

Sorry about your luck ;).

kmac
  • 688
  • 4
  • 15
0

The issue is that you are re-implementing the shelve module without implementing object writeback.

When:

self.b = dict()

occurs the __setattr__ stores a pickled dictionary in the cache:

Then when:

self.b["foo"] = "bar"

The __getattribute__ unpickles the dictionary and returns it. Then Python sets that dictionary's foo key to bar. Then it throws away the dictionary.

Dan D.
  • 73,243
  • 15
  • 104
  • 123