4

I have a class or extension type written in cython like this:

cdef class Self_Organized_Map:
     cdef 
     def __cinit__(self,np.ndarray data,.....):
     ....

I created a python module from this cython file (which its name is som.pyx) using distutils , then I import this in python and used it to create and train a model , but when I want to save my model using pickle it gives me this err :

TypeError: can't pickle som.Self_Organized_Map objects

what is wrong with pickle or my code ? does pickle cant save extension object ?

Cœur
  • 37,241
  • 25
  • 195
  • 267
Javad Sameri
  • 1,218
  • 3
  • 17
  • 30
  • 1
    There's quite a bit of documentation and previous questions that it doesn't look like you've read: such as https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled ([extension types don't define a `__dict__` by default}(http://cython.readthedocs.io/en/latest/src/reference/extension_types.html#attributes)). A few relevant previous questions: http://stackoverflow.com/questions/12646436/pickle-cython-class http://stackoverflow.com/questions/36301322/pickle-cython-class-with-c-pointers – DavidW May 03 '17 at 13:38
  • 1
    Also, your title mentions pointers, but your code doesn't show any. Perhaps clarify? – DavidW May 03 '17 at 13:39
  • @DavidW thanks I corrected it ! – Javad Sameri May 04 '17 at 19:12

2 Answers2

5

Cython classes aren't pickleable by default, so you need to implement the Pickle interface yourself. There's a number of different levels you can do this on but __getstate__ and __setstate__ is the most user-friendly level so it's a good place to start unless you have good reasons otherwise.

Where the contents of the class are pickleable it's as easy as returning a tuple of them in __getstate__ and the reverse operation in __setstate__. Memoryviews themselves aren't pickleable, but have a base attribute that may be.

cdef class C:
    cdef double[:] array
    cdef python_obj
    cdef int integer

    def __init__(self,array,python_obj,integer):
        self.array = array
        self.python_obj = python_obj
        self.integer = integer

    def __getstate__(self):
        return (self.array.base, # memoryviews aren't pickleable, need to get underlying object
                          # and hope it's pickleable
                self.python_obj, self.integer)

    def __setstate__(self,x):
        self.array, self.python_obj, self.integer = x

If your class holds a C or C++ object then it's much more complicated. For simple types good places to start are just copy the memory into a bytearray or to take advantage of Cython's default struct<->dict interconversion. However if the class contains pointers then this won't work and you need to implement a reliable load/save mechanism for it in C/C++.

DavidW
  • 29,336
  • 6
  • 55
  • 86
4

Since Cython 0.26 (released July 2017), cdef classes can automatically be pickled, as long as they do not contain pointers or unions. For classes containing structs, automatic pickling can be enabled with the @cython.auto_pickle(True) decorator. It is disabled by default due to high code overhead an other reasons.

More information can be found in the changelog and on the website of Stefan Behnel.

m00am
  • 5,910
  • 11
  • 53
  • 69