0

I'm working on a decorator to implement some behaviors for an immutable class. I'd like a class to inherit from namedtuple (to have attribute immutability) and also want to add some new methods. Like this ... but correctly preventing new attributes being assigned to the new class.

When inheriting from namedtuple, you should define __new__ and set __slots__ to be an empty tuple (to maintain immutability):

def define_new(clz):
    def __new(cls, *args, **kwargs):
        return super(clz, cls).__new__(cls, *args, **kwargs)

    clz.__new__ = staticmethod(__new) # delegate namedtuple.__new__ to namedtuple
    return clz

@define_new
class C(namedtuple('Foo', "a b c")):
    __slots__ = () # Prevent assignment of new vars
    def foo(self): return "foo"

C(1,2,3).x = 123 # Fails, correctly

... great. But now I'd like to move the __slots__ assignment into the decorator:

def define_new(clz):
    def __new(cls, *args, **kwargs):
        return super(clz, cls).__new__(cls, *args, **kwargs)

    #clz.__slots__ = ()
    clz.__slots__ = (123) # just for testing

    clz.__new__ = staticmethod(__new)
    return clz

@define_new
class C(namedtuple('Foo', "a b c")):
    def foo(self): return "foo"

c = C(1,2,3)
print c.__slots__ # Is the (123) I assigned!
c.x = 456         # Assignment succeeds! Not immutable.
print c.__slots__ # Is still (123)

Which is a little surprising.

Why has moving the assignment of __slots__ into the decorator caused a change in behavior?

If I print C.__slots__, I get the object I assigned. What do the x get stored?

user48956
  • 14,850
  • 19
  • 93
  • 154
  • 1
    I'd imagine this would have to be done in a metaclass. The class body is executed before the class object exists, your decorator merely adds the attribute afterwards, but by then it is too late. Is the only reason you are inheriting from namedtuple to not allow assignment? – juanpa.arrivillaga Feb 01 '18 at 17:16
  • 1
    What made you think defining `__new__` was necessary? – user2357112 Feb 01 '18 at 17:24
  • https://stackoverflow.com/questions/42385916/inheriting-from-a-namedtuple-base-class-python – user48956 Feb 01 '18 at 17:30
  • That's only if you want to customize instance initialization; if you want to customize that, you have to define `__new__` instead of `__init__`. – user2357112 Feb 01 '18 at 17:32
  • Yeah -- turns out you're right. The issue with slots remains though... – user48956 Feb 01 '18 at 17:34
  • @user48956: When you do get a non-empty `__slots__` class attribute created at the appropriate time, Python will not allow the class to be created: `nonempty __slots__ not supported for subtype of 'tuple'`: https://stackoverflow.com/questions/44725110/why-cant-nonempty-slots-be-used-with-int-tuple-bytes-subclasses#comment84133439_44725110 – Blender Feb 01 '18 at 17:39
  • Hi, I'm curious if my answer actually helped you with the question? – user4815162342 Feb 15 '18 at 14:12
  • It did, and thank you. It led to a lot of refactoring into hierarchies if immutable object classes – user48956 Feb 15 '18 at 16:15

2 Answers2

5

The code doesn't work because __slots__ is not a normal class property consulted at run-time. It is a fundamental property of the class that affects the memory layout of each of its instances, and as such must be known when the class is created and remain static throughout the its lifetime. While Python (presumably for backward compatibility) allows assigning to __slots__ later, the assignment has no effect on the behavior of existing or future instances.

How __slots__ is set

The value of __slots__ determined by the class author is passed to the class constructor when the class object is being created. This is done when the class statement is executed; for example:

class X:
    __slots__ = ()

The above statement is equivalent1 to creating a class object and assigning it to X:

X = type('X', (), {'__slots__': ()})

The type object is the metaclass, the factory that creates and returns a class when called. The metaclass invocation accepts the name of the type, its superclasses, and its definition dict. Most of the contents of the definition dict can also be assigned later The definition dict contains directives that affect low-level layour of the class instances. As you discovered, later assignment to __slots__ simply has no effect.

Setting __slots__ from the outside

To modify __slots__ so that it is picked up by Python, one must specify it when the class is being created. This can be accomplished with a metaclass, the type responsible for instantiating types. The metaclass drives the creation of the class object and it can make sure __slots__ makes its way into the class definition dict before the constructor is invoked:

class DefineNew(type):
    def __new__(metacls, name, bases, dct):

        def __new__(cls, *new_args, **new_kwargs):
            return super(defcls, cls).__new__(cls, *new_args, **new_kwargs)

        dct['__slots__'] = ()
        dct['__new__'] = __new__

        defcls = super().__new__(metacls, name, bases, dct)
        return defcls

class C(namedtuple('Foo', "a b c"), metaclass=DefineNew):
    def foo(self):
        return "foo"

Testing results in the expected:

>>> c = C(1, 2, 3)
>>> c.foo()
'foo'
>>> c.bar = 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'C' object has no attribute 'bar'

Metaclass mixing pitfall

Note that the C type object will itself be an instance of DefineMeta - which is not surprising, since that follows from the definition of a metaclass. But this might cause an error if you ever inherit from both C and a type that specifies a metaclass other than type or DefineMeta. Since we only need the metaclass to hook into class creation, but are not using it later, it is not strictly necessary for C to be created as an instance of DefineMeta - we can instead make it an instance of type, just like any other class. This is achieved by changing the line:

        defcls = super().__new__(metacls, name, bases, dct)

to:

        defcls = type.__new__(type, name, bases, dct)

The injection of __new__ and __slots__ will remain, but C will be a most ordinary type with the default metaclass.

In conclusion...

Defining a __new__ which simply calls the superclass __new__ is always superfluous - presumably the real code will also do something different in the injected __new__, e.g. provide the default values for the namedtuple.


1 In the actual class definition the compiler adds a couple of additional items to the class dict, such as the module name. Those are useful, but they do not affect the class definition in the fundamental way that __slots__ does. If X had methods, their function objects would also be included in the dict keyed by function name - automatically inserted as a side effect of executing the def statement in the class definition namespace dict.
user4815162342
  • 141,790
  • 18
  • 296
  • 355
2

__slots__ has to be present during class creation. It affects the memory layout of a class's instances, which isn't something you can just change at will. (Imagine if you already had instances of the class and you tried to reassign the class's __slots__ at that point; instances would all break.) The processing that bases the memory layout on __slots__ only happens during class creation.

Assigning __slots__ in a decorator is too late to do anything. It has to happen before class creation, in the class body or a metaclass __new__.

Also, your define_new is pointless; namedtuple.__new__ already does what you need your __new__ to do.

user2357112
  • 260,549
  • 28
  • 431
  • 505