Mixin Cython class and SqlAlchemy

Question

Abstract:

I have one cython class which represents a business unit. This class is declared in pure cython style.

In one project, I need to map the business unit to a database. For doing this I would like to import the .pxd file and "map" it with SQLAlchemy.

Cython definition

Let's suppose the class Equipment. The class is defined the .pyx and the class interface in a .pxd (because I need to cimport it in other module).

equipment.pxd

cdef class Equipment:
    cdef readonly int x
    cdef readonly str y

equipment.pyx

cdef class Equipment:
    def __init__(self, int x, str y):
        self.x = x
        self.y = y

I compile everything and get a equipment.pyd file. So far, it's ok. This file contains the business logic model and must not be altered.

Mapping

Then in one application, I import the equipment.pyd and map it with SQLAlchemy.

from sqlalchemy import Table, Column, Integer, String
from sqlalchemy.orm import mapper
from equipment import Equipment

metadata = MetaData()

# Table definition
equipment = Table(
    'equipment', metadata,
    Column('id', Integer, primary_key=True),
    Column('x', Integer),
    Column('y', String),
)

# Mapping the table definition with the class definition
mapper(Equipment, equipment)

TypeError: can't set attributes of built-in/extension type 'equipment.Equipment'

Indeed, SQLAlchemy is trying to create the Equipment.c.x, Equipment.c.y, ... Which is not possible in Cython because it is not defined in the .pxd...

So how can I map Cython class to SQLAlchemy?

Not satisfying solution

If I define the equipment class in python mode in .pyx file, it's working because at the end, it's only 'python' object in cython class definition.

equipment.pyx

class Equipment:
    def __init__(self, x, y):
        self.x = x
        self.y = y

But I'm losing at lot of functionnalities which is why I need pure Cython.

Thank you! :-)

-- EDIT PART --

Semi satisfying solution

Keep the .pyx and .pxd files. Inherit from the .pyd. Try to Map.

mapping.py

from sqlalchemy import Table, Column, Integer, String
from sqlalchemy.orm import mapper
from equipment import Equipment

metadata = MetaData()

# Table definition
equipment = Table(
    'equipment', metadata,
    Column('id', Integer, primary_key=True),
    Column('x', Integer),
    Column('y', String),
)

# Inherit Equipment to a mapped class
class EquipmentMapped(Equipment):
    def __init__(self, x, y):
        super(EquipmentMapped, self).__init__(x, y)

# Mapping the table definition with the class definition
mapper(EquipmentMapped, equipment)

from mapping import EquipmentMapped

e = EquipmentMapped(2, 3)

print e.x

## This is empty!

In order to make it work, I must define each attributes as a property!

equipment.pxd

cdef class Equipment:
    cdef readonly int _x
    cdef readonly str _y

equipment.pyx

cdef class Equipment:
    def __init__(self, int x, str y):
        self.x = x
        self.y = y
    property x:
        def __get__(self):
            return self._x
        def __set__(self, x):
            self._x = x
    property y:
        def __get__(self):
            return self._y
        def __set__(self, y):
            self._y = y

This is not satisfying because :lazy_programmer_mode on: I have a lot of changes to do in the business logic... :lazy_programmer_mode off:

Answer 1

I think the basic problem is that when you call mapper it does (among other things)

Equipment.x = ColumnProperty(...) # with some arguments
Equipment.y = ColumnProperty(...)

when ColumnProperty is a sqlalchemy defined property, so when you do e.x = 5, it can notice that the value has changed at do all the database related stuff around it.

The obviously doesn't play along nicely with the Cython class below that you're trying to use to control the storage.

Personally, I suspect the only real answer to to define a wrapper class that holds both the Cython class and the sqlalchemy mapped class, and intercepts all attribute accesses and method calls to keep them in sync. A rough implementation is below, which should work for simple cases. It is barely tested though, so almost certainly has bugs and corner cases that it misses. Beware!

def wrapper_class(cls):
    # do this in a function so we can create it generically as needed
    # for each cython class
    class WrapperClass(object):
        def __init__(self,*args,**kwargs):
            # construct the held class using arguments provided
            self._wrapped = cls(*args,**kwargs)

        def __getattribute__(self,name):
            # intercept all requests for attribute access.
            wrapped = object.__getattribute__(self,"_wrapped")
            update_from = wrapped
            update_to = self
            try:
                o = getattr(wrapped,name)
            except AttributeError:
                # if we can't find it look in this class instead.
                # This is reasonable, because there may be methods defined
                # by sqlalchemy for example
                update_from = self
                update_to = wrapped
                o = object.__getattribute__(self,name)
            if callable(o):
                return FunctionWrapper(o,update_from,update_to)
            else:
                return o

        def __setattr__(self,name,value):
            # intercept all attempt to write to attributes
            # and set it in both this class and the wrapped Cython class
            if name!="_wrapped":
                try:
                    setattr(self._wrapped,name,value)
                except AttributeError:
                    pass # ignore errors... maybe bad!
            object.__setattr__(self,name,value)
    return WrapperClass

class FunctionWrapper(object):
    # a problem we have is if we call a member function.
    # It's possible that the member function may change something
    # and thus we need to ensure that everything is updated appropriately
    # afterwards
    def __init__(self,func,update_from,update_to):
        self.__func = func
        self.__update_from = update_from
        self.__update_to = update_to

    def __call__(self,*args,**kwargs):
        ret_val = self.__func(*args,**kwargs)

        # for both Cython classes and sqlalchemy mappings
        # all the relevant attributes exist in the class dictionary
        for k in self.__update_from.__class__.__dict__.iterkeys():
            if not k.startswith('__'): # ignore private stuff
                try:
                    setattr(self.__update_to,k,getattr(self.__update_from,k))
                except AttributeError:
                    # There may be legitmate cases when this fails
                    # (probably relating to sqlalchemy functions?)
                    # in this case, replace raise with pass
                    raise
        return ret_val

To use it you'd do something like:

class EquipmentMapped(wrapper_class(Equipment)):
    # you may well have to define __init__ here
    # you'll have to check yourself, and see what sqlalchemy does...
    pass

mapper(EquipmentMapped,equipment)

Please remember this is a horrendous workround that basically just duplicates all your data in two places then desperately tries to keep in in sync.

---

Edit: The original version of this gave a mechanism to automated a line of enquiry OP had tried but decided was a lot of effort to do manually (defining properties on the Cython class, which only succeeds in overriding sqlalchemy's mechanisms for tracking changes) Further testing confirms that it doesn't work. If you're curious as to what not to do then look at the edit history!