When does Python fall back onto class __dict__ from instance __dict__?

Question

Please see the below snippet:

class Foo:
    class_var = "hi"

foo = Foo()
assert foo.class_var is Foo.class_var
assert "class_var" in Foo.__dict__
assert "class_var" not in foo.__dict__

All assertions here pass, though I am not sure if it's surprising that the identity assertion passes.

When and how does Python fall back onto a class __dict__ from an instance __dict__?

Answer 1

As far as I know when foo.class_var is called the following steps happen:

• Python starts to look for class_var in the namespace of the foo object.
• If it finds it, it returns it.
• In this case however it doesn't find it so it looks in the type of foo, which is Foo.
• It finds it in Foo and returns it.

Answer 2

According to (already mentioned) [Python.Docs]: Data model (emphasis is mine):

Custom classes

Custom class types are typically created by class definitions (see section Class definitions). A class has a namespace implemented by a dictionary object. Class attribute references are translated to lookups in this dictionary, e.g., C.x is translated to C.__dict__["x"] (although there are a number of hooks which allow for other means of locating attributes). When the attribute name is not found there, the attribute search continues in the base classes.

...

Class instances

A class instance is created by calling a class object (see above). A class instance has a namespace implemented as a dictionary which is the first place in which attribute references are searched. When an attribute is not found there, and the instance’s class has an attribute by that name, the search continues with the class attributes.

...

Invoking Descriptors

...

The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance, a.x has a lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'], and continuing through the base classes of type(a) excluding metaclasses.

However, if the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead. Where this occurs in the precedence chain depends on which descriptor methods were defined and how they were called.

Attributes defined inside a class definition (but outside the initializer (or other methods)) are called class attributes, and are bound to the class itself rather than its instances. It's like static members from C++ or Java. [Python.Docs]: Compound statements - Class definitions states (emphasis still mine):

Programmer’s note: Variables defined in the class definition are class attributes; they are shared by instances. Instance attributes can be set in a method with self.name = value. Both class and instance attributes are accessible through the notation “self.name”, and an instance attribute hides a class attribute with the same name when accessed in this way. Class attributes can be used as defaults for instance attributes, but using mutable values there can lead to unexpected results. Descriptors can be used to create instance variables with different implementation details.

So, the attribute lookup order can be summarized like below (traverse in ascending order, when attribute name found simply return its value (therefore ignoring the remaining entries)). The first steps performed by the (builtin) __getattribute__ method:

0. Descriptors (if any - note that their presence could also be triggered indirectly (by other features))

1. Instance namespace (foo.__dict__)

2. Instance class namespace (Foo.__dict__)

3. Instance class base classes namespaces (e.__dict__ for e in Foo.__mro__)

4. Anything that a custom __getattr__ method might return

The above is what typically happens, as Python being highly customizable that can be altered (e.g. __slots__).

For an exact behavior, you could check the source code ([GitHub]: python/cpython - (main) cpython/Objects):

• typeobject.c: type_getattro (optionally: super_getattro, slot_tp_getattro)

• object.c: _PyObject_GenericGetAttrWithDict

Here's an example that will clear things up (hopefully).

code00.py:

#!/usr/bin/env python

import sys
from pprint import pformat as pf


def print_dict(obj, header="", indent=0, filterfunc=lambda x, y: not x.startswith("__")):
    if not header:
        header = getattr(obj, "__name__", None)
    if header:
        print("{:}{:}.__dict__:".format("  " * indent, header))
    lines = pf({k: v for k, v in getattr(obj, "__dict__", {}).items() if filterfunc(k, v)}, sort_dicts=False).split("\n")
    for line in lines:
        print("{:}{:}".format("  " * (indent + 1), line))
    print()


class Descriptor:
    def __init__(self, name):
        self.name = name

    def __get__(self, instance, cls):
        print("{:s}.__get__".format(self.name))

    def __set__(self, instance, value):
        print("{:s}.__set__ - {:}".format(self.name, value))

    def __delete__(self, instance):
        print("{:s}.__delete__".format(self.name))


class Demo:
    cls_attr0 = 3.141593
    cls_attr1 = Descriptor("cls_attr1")

    '''
    def __getattribute__(self, name):
        print("__getattribute__:", self, name)
        return super().__getattribute__(name)
    '''

    '''
    def __getattr__(self, name):
        print("__getattr__:", self, name)
        return "something dummy"
    '''

    def __init__(self):
        self.inst_attr0 = 2.718282


def main(*argv):
    print("ORIGINAL")
    demos = [Demo() for _ in range(2)]
    demo0 = demos[0]
    demo1 = demos[1]
    print_dict(Demo)
    print_dict(demo0, header="demo0")
    print("\ndemo0 attrs:", demo0.cls_attr0, demo0.cls_attr1, demo0.inst_attr0)
    print_dict(demo1, header="\ndemo1")
    print("\ndemo1 attrs:", demo1.cls_attr0, demo1.cls_attr1, demo1.inst_attr0)

    print("\nALTER 1ST INSTANCE OBJECT")
    demo0.inst_attr0 = -3
    demo0.cls_attr0 = -5

    print_dict(Demo)
    print_dict(demo0, header="demo0")
    print("\ndemo0 attrs:", demo0.cls_attr0, demo0.cls_attr1, demo0.inst_attr0)
    print_dict(demo1, header="\ndemo1")
    print("\ndemo1 attrs:", demo1.cls_attr0, demo1.cls_attr1, demo1.inst_attr0)

    print("\nALTER CLASS")
    Demo.cls_attr0 = -7
    Demo.cls_attr1 = -9
    print_dict(Demo, header="Demo")
    print_dict(demo1, header="demo0")
    print("\ndemo0 attrs:", demo0.cls_attr0, demo0.cls_attr1, demo0.inst_attr0)
    print_dict(demo1, header="\ndemo1")
    print("\ndemo1 attrs:", demo1.cls_attr0, demo1.cls_attr1, demo1.inst_attr0)


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q072399556]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" code00.py
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32

ORIGINAL
Demo.__dict__:
  {'cls_attr0': 3.141593,
   'cls_attr1': <__main__.Descriptor object at 0x00000171B0B24FD0>}

demo0.__dict__:
  {'inst_attr0': 2.718282}

cls_attr1.__get__

demo0 attrs: 3.141593 None 2.718282

demo1.__dict__:
  {'inst_attr0': 2.718282}

cls_attr1.__get__

demo1 attrs: 3.141593 None 2.718282

ALTER 1ST INSTANCE OBJECT
Demo.__dict__:
  {'cls_attr0': 3.141593,
   'cls_attr1': <__main__.Descriptor object at 0x00000171B0B24FD0>}

demo0.__dict__:
  {'inst_attr0': -3, 'cls_attr0': -5}

cls_attr1.__get__

demo0 attrs: -5 None -3

demo1.__dict__:
  {'inst_attr0': 2.718282}

cls_attr1.__get__

demo1 attrs: 3.141593 None 2.718282

ALTER CLASS
Demo.__dict__:
  {'cls_attr0': -7, 'cls_attr1': -9}

demo0.__dict__:
  {'inst_attr0': 2.718282}


demo0 attrs: -5 -9 -3

demo1.__dict__:
  {'inst_attr0': 2.718282}


demo1 attrs: -7 -9 2.718282

Done.

Answer 3

You should know the difference between a "class attribute" and an "instance attribute", in your example, you have class_var which is class attribute, in the first assert statement, it check if class_var is in the instance of Foo , it pass, that because foo is of type Foo let's try another more clear examples here:

class Foo:
    var1 = "Hello"
    def __init__(self,var2):
        self.var2 = var2
foo = Foo("Hello")
assert foo.var1 is Foo.var1
assert "var1" in Foo.__dict__
assert "var2" in foo.__dict__
assert "var1" in foo.__dict__

notice something, the last assert statement is gonna raise an error, because var1 here is a class attribute, not instance attribute.

Answer 4

I like to explain it by an example becase people like their own experience. Example with a subclass:

class A:
    x = 1
    w = -1


class B(A):
    y = 2
    w = -2

    def __init__(self):
        self.z = 3
        self.w = -3


def get_dict(obj):
    """Get a __dict__ only with normal user defined keys"""
    return {k: v for k, v in obj.__dict__.items() if not k.startswith('__')}


b = B()

You can imagine that getattr(b, name) is implemented simplified like:

if name in b.__dict__:
    return b.__dict__[name]
elif name in B.__dict__:
    return B.__dict__[name]
elif name in A.__dict__:
    return A.__dict__[name]
else:
    raise AttributeError(f"'B' object has no attribute '{name}'")

Explore all __dict__ of the instance and all classes:

>>> get_dict(A)
{'x': 1, 'w': -1}
>>> get_dict(B)
{'y': 2, 'w': -2}
>>> get_dict(b)
{'z': 3, 'w': -3}

# change the values
>>> A.w = 10
>>> B.w = 20
>>> b.w = 30

>>> A.x, B.y, b.z
(1, 2, 3)
>>> A.w, B.w, b.w
(10, 20, 30)

>>> get_dict(A)
{'x': 1, 'w': 10}
>>> get_dict(B)
{'y': 2, 'w': 20}
>>> get_dict(b)
{'z': 3, 'w': 30}

The CristiFat's answer is precise, but maybe not easy enough for everybody.