TypeError when extending a class with additional attributes

Question

I want to extend the standard python bytes class with some additional attributes, so that I can handle them like any normal bytes objext (e.g. putting them into a list and sort them).

Thus I created my own class that inherits from bytes and overrode the constructor to take additional attributes, setting them before invoking the parent class (bytes) constructor.

class inheritest(bytes):
    def __init__(self, bs: bytes, x: int = 0):
        print("child class init")
        self.x = x
        super().__init__(bs)


print(inheritest(b'foobar', 3))

This approach does not work: I get a type error about a wrong argument signature being passed to the bytes constructor, although I only invoke it in line 5 with a single argument of the type bytes, which should be fine.
Even more, note that the print statement is never executed, so the constructor of the inheritest class is never executed but the argument type signature check, raising the TypesError, seems to happen beforehand.

Traceback (most recent call last):
  File "inheritest.py", line 8, in <module>
    print(inheritest(b'foobar', 3))
TypeError: bytes() argument 2 must be str, not int

So what am I doing wrong about inheritance and attribute extension?

hiro protagonist · Answer 1 · 2019-07-02T14:15:23.753

you need to override __new__ in order to inherit from bytes:

class MyBytes(bytes):
    def __new__(cls, *args, **kwargs):
        self = super().__new__(cls, *args, *kwargs)
        return self

now to extend as you would like i suggest this:

class MyBytes(bytes):
    def __new__(cls, source, x):
        self = super().__new__(cls, source)
        self.x = x
        return self

    def __str__(self):
        return f"<{repr(self)}, {self.x}>"


b = MyBytes(b"", 3)
print(b)  # <b'', 3>
b = MyBytes(b"345", 5)
print(b)  # <b'345', 5>

note:

i ignored the other arguments bytes would accept (encoding and errors)
__str__ only works as long as you do not override __repr__.
instances of MyByte are mutable now; depending on what you need you may either avoid that x can be changed (with a property) and set the __slots__ attribute for the class - or provide a __hasĥ__ method that takes x into account.

this would also work:

class MyBytes(bytes):
    def __new__(cls, source, x):
        self = super().__new__(cls, source)
        return self

    def __init__(self, source, x):
        self.x = x

@ReblochonMasque i'd say along the lines of this: `bytes` is immutable; therfore it is too late to initialize an instance when `__init__` is called. — hiro protagonist, Jul 02 '19 at 11:04
oh, this answer may be more to the point: https://stackoverflow.com/a/2673863/4954037 — hiro protagonist, Jul 02 '19 at 11:05

MisterMiyagi · Accepted Answer · 2019-07-02T12:59:06.030

TLDR: You are incorrectly overwriting __init__ instead of __new__.

class XBytes(bytes):
    __slots__ = 'x',  # avoid arbitrary attributes

    def __new__(cls, bs: bytes, x: int = 0):
        # call original __new__ with expected signature
        self = super().__new__(cls, bs)
        self.x = x
        return self

Python types canonically have two methods that are used when constructing an object:

__new__ creates the object ("constructor")
__init__ creates the state ("initialiser")

Notably, both are called when creating an object. You can think of constructing an instance of Class (e.g. Class(1, 2, 3)) as equivalent to this:

def new(cls, *args, **kwargs):
    """same as ``cls(*args, **kwargs)``"""
    self = cls.__new__(cls, *args, **kwargs)
    if isinstance(self, cls):
       cls.__init__(self, *args, **kwargs)

Note how __new__ creates an object, while __init__ only changes it. For immutable types, you must override __new__ since they cannot be changed.

If you override __init__, the signature of __new__ is not changed! When calling inheritest(b'foobar', 3), the arguments b'foobar', 3 are passed to the original bytes.__new__. This happens before your custom __init__ is called, thus never triggering print.

Thank you. Could it be that `__new__` is usually left out in all literature teaching python? Never heard of its role in object creation before. — schmittlauch, Jul 02 '19 at 15:29
Teaching material generally focuses only on subclassing ``object``, since it is the most common. The more complex parts of the Python object model are generally ignored. You can find more information in the docs (https://docs.python.org/3/reference/datamodel.html#object.__new__) and there are some tutorials around if you specifically look for ``__new__``. — MisterMiyagi, Jul 02 '19 at 15:34

TypeError when extending a class with additional attributes

2 Answers2