Cython and deepcopy() woes with referenced methods/functions. Any alternative ideas?

Question

I've been playing with Cython recently for the speed ups, but my project inherits a module that has a copy() method which uses deepcopy(). I tried implementing the deepcopy() within an overrided version of copy(), and I thought I had it working, but it doesn't appear to be anymore.

TypeError: object.__new__(cython_binding_builtin_function_or_method) is not safe,
   use cython_binding_builtin_function_or_method.__new__()

This is occuring in python/lib/copy_reg.py here:

return cls.__new__(cls, *args)

I'm on Python 2.7 here. Is it possible that a newer version of Python returns from deepcopy() in a "safe" way? I'm also on the latest version of Cython, 0.15.1.

Update3

Note that I've removed the previous updates to keep this as simple as possible.

Ok! I think I found the incompatibility but I don't really know what to do about it.

class CythonClass:
    def __init__(self):
        self._handle = self._handles.get("handle_method")

    def call_handle(self):
        self._handle(self)

    def handle_method(self):
        print "I'm a little handle!"

    handles = {"handle_method", handle_method}

Then in my main app:

from cython1 import CythonClass
from copy import deepcopy

if __name__ == "__main__":
    gc1 = CythonClass()
    gc1.call_handle()
    gc2 = deepcopy(gc1)

I get:

I'm a little handle!

Traceback (most recent call last):
  File "cythontest.py", line 8, in <module>
    gc2 = deepcopy(gc1)
  File "C:\python26\lib\copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "C:\python26\lib\copy.py", line 292, in _deepcopy_inst
    state = deepcopy(state, memo)
  File "C:\python26\lib\copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "C:\python26\lib\copy.py", line 255, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\python26\lib\copy.py", line 189, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "C:\python26\lib\copy.py", line 323, in _reconstruct
    y = callable(*args)
  File "C:\python26\lib\copy_reg.py", line 93, in __newobj__
    return cls.__new__(cls, *args)
TypeError: object.__new__(cython_binding_builtin_function_or_method) is not safe, use cython_binding_builtin_function_or_method.__new__()

The key is the function/handle reference:

handles = {"handle_method", handle_method}

If I don't include the method/function reference, Cython will not blow up during deepcopy. If I include one, it doesn't like how deepcopy/copy_reg copies the reference over.

Any ideas besides not using method/function references? I have a bit of untangling to do if that the simple answer. (which I'm already working on as I finish typing this)

Thanks!

Answer 1

found this:

"Does deepcopy work properly with Cython?"

No. In this case (you are using extension types, i.e cdef classes) you have to implement the pickle protocol for your class http://docs.python.org/library/pickle.html#pickling-and-unpickling-extension-types

from here: https://groups.google.com/forum/#!topic/cython-users/p2mzJrnOH4Q

"implementing the pickle protocol" in the linked article is actually simple, and solved my problems trivially (although I am doing things slightly differently - my class is a cdef class, and I have a pointer to a CPP object stored there which cannot be trivially duplicated - I don't know if this will solve the python-inheritance problem above, but it is certainly worth a try.)

Anyway, implementing the pickle protocol is trivial (the example below is using "C++ cython", which has a double meaning for the del keyword, among other things.):

cdef class PyObject(object):
    cdef CppObject* cpp
    cdef object arg1
    cdef object arg2

    def __cinit__(self, arg1=[], arg2=False):
        # C++ constructor using python values, store result in self.cpp.

        # new code: cache the python arguments that were used.
        self.arg1 = arg1
        self.arg2 = arg2

    def __init__(self, arg1=[], arg2=False):
        # logic for validating arguments.
        pass

    def __dealloc__(self):
        if not self.cpp == NULL:
            del self.cpp

    def __reduce__(self):
        # a tuple as specified in the pickle docs - (class_or_constructor, 
        # (tuple, of, args, to, constructor))
        return (self.__class__, (self.arg1, self.arg2))

When I try this, I can call copy.deepcopy() on a dict containing an instance of my Cython extension type, and get a new dictionary containing a new instance (with a different memory address when printed to terminal.) Previously the same code caused a segfault.

Answer 2

Ah, finally found the "Answer your own question" button.

I'm probably being impatient, but since nobody has replied yet (I mean who's using Cython and answering questions on a Thursday afternoon), I thought I would close this one off.

1) Cython doesn't like deepcopy on Classes which have function/method referenced variables. Those variable copies will fail. From what I can tell, there's no working around it, you just have to come up with an new design that doesn't require them. I ended up doing so with the same code above, and in my project.

2) Cython doesn't handle the property decorators at all. You can't @property and @<property name>.setter. Properties need to be set the old fashion way. e.g. <property name> = property(get_property, set_property).

3) An inherited non-Cython class's methods "might" not be accessible. I know this vague. I can't completely explain it. All I'll say is I was inheriting NetworkX.DiGraph, and number_of_nodes() was no accessible after Cython, when it was straight Python. I had to create a reference to the method and use it. e.g. number_of_verts = NetworkX.DiGraph.number_of_nodes.