Python Multiple inheritance metaclass error

Question

I am trying inherit from ABC and ctypes.Structure but I'm getting multiple inheritance and metaclass errors.

What am I doing wrong?

My code is as follows:

from ctypes import *
from abc import ABC, abstractmethod, ABCMeta

class FinalMeta(ABCMeta, type(Structure)):
    def __new__(mcs, name, bases, namespace, **kwargs):
        print("FinalMeta.__new__(mcs, name, bases, namespace, **kwargs)")
        print("  mcs =", mcs)
        print("  name =", name)
        print("  bases =", bases)
        print("  namespace =", namespace)
        print("  kwargs =", kwargs)
        cls = super().__new__(mcs, name, bases, namespace, **kwargs)
        print("<-- cls =", cls)
        print()
        return cls


class MessageBase(ABC, Structure, metaclass=FinalMeta):
    def __init__(self, *args, **kwargs):
        super().__init__()
    
    @property
    @abstractmethod
    def format(self):
        pass

Error:

File "D:\Development\messages.py", line 13, in __new__
    cls = super().__new__(mcs, name, bases, namespace, **kwargs)
  File "C:\Users\AppData\Local\Programs\Python\Python37-32\lib\abc.py", line 126, in __new__
    cls = super().__new__(mcls, name, bases, namespace, **kwargs)
TypeError: _ctypes.PyCStructType.__new__(FinalMeta) is not safe, use type.__new__()**

user2357112 · Answer 1 · 2020-07-13T05:08:17.020

This isn't going to work. yAzou's answer might look like it works, but it doesn't - it skips crucial initialization and leaves MessageBase in a broken state, the most obvious symptom being that even if you fill in _fields_ and implement the abstract property in a concrete subclass, you still won't be able to create instances.

Trying to do multiple inheritance with types written in C isn't well supported. Part of the behavior might be a Python bug, but I doubt it'll ever work well. Here's what's going on.

_ctypes.PyCStructType is the metaclass of ctypes.Structure. This metaclass is written in C, which means its __new__ is generated as a wrapper around its C-level tp_new slot.

The tp_new wrapper wants to make sure you never skip past C-level initialization by doing something like object.__new__(tuple) (or worse, tuple.__new__(dict)), because that could cause memory corruption. For that reason, it has the following safety check:

static PyObject *
tp_new_wrapper(PyObject *self, PyObject *args, PyObject *kwds)
{
    ...
    /* Check that the use doesn't do something silly and unsafe like
       object.__new__(dict).  To do this, we check that the
       most derived base that's not a heap type is this type. */
    staticbase = subtype;
    while (staticbase && (staticbase->tp_new == slot_tp_new))
        staticbase = staticbase->tp_base;
    /* If staticbase is NULL now, it is a really weird type.
       In the spirit of backwards compatibility (?), just shut up. */
    if (staticbase && staticbase->tp_new != type->tp_new) {
        PyErr_Format(PyExc_TypeError,
                     "%s.__new__(%s) is not safe, use %s.__new__()",
                     type->tp_name,
                     subtype->tp_name,
                     staticbase->tp_name);
        return NULL;
    }

This tries to make sure that when you call ClassA.__new__(ClassB, ...) with ClassA written in C, ClassA is the C class furthest down ClassB's inheritance hierarchy. However, it follows a single chain of base classes through the tp_base pointers, ignoring multiple inheritance!

With the way tp_base pointers are initialized (see best_base in Objects/typeobject.c), following tp_base pointers would usually lead to the most derived C class, but this depends on each C class adding fields to its (single) base class's instance memory layout. _ctypes.PyCStructType doesn't do that (it replaces the class dict with a weird custom dict subclass to store its data instead), so the tp_base chain for FinalMeta goes FinalMeta->ABCMeta->type->object.

Since _ctypes.PyCStructType isn't in the tp_base chain, tp_new_wrapper thinks type is the most derived C ancestor of FinalMeta, and it thinks you should be calling type.__new__ instead. If you call type.__new__, though, you'll skip the initialization this check was supposed to stop you from skipping, leaving your class in a broken state.

You might think you could fix this by rearranging the bases to put type(Structure) ahead of ABCMeta, but that only solves half the problem. You'll be able to define your MessageBase class, but ABCMeta.__new__ won't run, and it won't do the initialization needed to mark your MessageBase abstract.

Also, there's an entirely unrelated, long-standing issue where even if MessageBase was marked abstract, you'd still be able to create instances of it. The check that prevents instantiating abstract classes is in object.__new__, and C classes other than object (and descendants of such classes) don't call object.__new__.

Python Multiple inheritance metaclass error

1 Answers1