Cython - Exposing C++ (vector and non-vector) objects returned by C++ function to Python

Question

I am working on a project where I have a large C++ code base available which I want to wrap with cython and made available in python.

In doing so, I am facing a situation where some of my C++ functions returns Vector object or simple object (with multiple attributes). I want to return this object to Python so that its values can be accessed.

For doing that, I am following this post almost exactly: How to expose a function returning a C++ object to Python without copying the object?

I have a very similar requirement. Please refer to the move construct that is implemented/used in the above forum.

Following is my code that I am trying to implement for simple (non-vector case):

test_header.pxd

from libcpp.vector cimport vector
from libcpp.string cimport string
from libcpp.map cimport map

cdef extern from "main_class.h":

    cdef cppclass main_class:
        int ID1
        double ID2


cdef extern from "class2.h":

    cdef cppclass class2:
        class2() except +
        class2(const double& T) except +
        void Add(const main_class& ev)
        const vector[main_class]& GetEvs() 

#Process Class
cdef extern from "Pclass.h":

    cdef cppclass Pclass:
        Pclass(const unsigned& n, const unsigned& dims) except +
        unsigned GetDims()        
        double processNext(const class2& data, const unsigned& num_iter) 


cdef extern from "main_algo.h":

    #TODO: Check if inheritance works correctly, virtual functions, objects, std::vector
    cdef cppclass main_algo:
        main_algo(const unsigned& dims) except +
        main_class getNext(Pclass& pr, const class2& d)

test.pyx

from header cimport main_class, class2, Pclass, main_algo
from libcpp.vector cimport vector
from libcpp.string cimport string
from libcpp.map cimport map


cdef extern from "<utility>":
    vector[class2]&& move(vector[class2]&&)
    main_class&& move(main_class&&)

cdef class main_class_2:
    cdef main_class* thisptr
    cdef main_class_3 evs

    def __cinit__(self,main_class_3 evs):
        self.evs = evs
        self.thisptr = &evs.evs

cdef class main_class_3:

    cdef main_class evs

    cdef move_from(self, main_class&& move_this):
        self.evs = move(move_this)


cdef class implAlgo:

    cdef:
        main_algo *_thisptr

    def __cinit__(implAlgo self):
        self._thisptr = NULL

    def __init__(implAlgo self, unsigned dims):
        self._thisptr = new main_algo(dims)

    def __dealloc__(implAlgo self):
        if self._thisptr != NULL:
            del self._thisptr


    cdef int _check_alive(implAlgo self) except -1:
        if self._thisptr == NULL:
            raise RuntimeError("Wrapped C++ object is deleted")
        else:
            return 0      

    cdef getNext(implAlgo self, Pclass& p, const class2& s):
        self._check_alive()
        cdef main_class evs = self._thisptr.getNext(p, sq)
        retval = main_class_3()
        retval.move_from(move(evs))

        return retval

Here, the class main_algo implements the method getNext() which returns an object of class main_class.

From test.pyx, I want to return this object to purely python file where its values can be accessed.

When I try to compile the above code, I get multiple instances of following error at several places wherever I use that method and I get similar error for different tokens like ')' or '*' . Some example errors are:

sources.cpp:5121:70: error: expected primary-expression before ‘*’ token
   __pyx_vtable_11Cython_test_7sources_main_class_3.move_from = (PyObject *(*)(struct __pyx_obj_11Cython_test_7sources_main_class_3 *, main_class &&))__pyx_f_11Cython_test_7sources_8main_class_3_move_from;
                                                                      ^
sources.cpp:5121:73: error: expected primary-expression before ‘)’ token
   __pyx_vtable_11Cython_test_7sources_main_class_3.move_from = (PyObject *(*)(struct __pyx_obj_11Cython_test_7sources_main_class_3 *, main_class &&))__pyx_f_11Cython_test_7sources_8main_class_3_move_from;
                                                                         ^
sources.cpp:5121:75: error: expected primary-expression before ‘struct’
   __pyx_vtable_11Cython_test_7sources_main_class_3.move_from = (PyObject *(*)(struct __pyx_obj_11Cython_test_7sources_main_class_3 *, main_class &&))__pyx_f_11Cython_test_7sources_8main_class_3_move_from;
                                                                           ^
sources.cpp:5121:133: error: expected primary-expression before ‘&&’ token
   __pyx_vtable_11Cython_test_7sources_main_class_3.move_from = (PyObject *(*)(struct __pyx_obj_11Cython_test_7sources_main_class_3 *, main_class &&))__pyx_f_11Cython_test_7sources_8main_class_3_move_from;

But all these tokens are related to the objects that I create to move the C++ object to python. There is no error with any other declaration.

Can someone please help to show me where am I wrong?

This may be useful: http://intermediate-and-advanced-software-carpentry.readthedocs.io/en/latest/c++-wrapping.html — Alexander, Jul 28 '16 at 19:14
Have you ensured that it's using c++11 mode (or higher) when it's compiled. For gcc you might add something like `extra_link_args=['-std=c++11']` to `setup.py` http://stackoverflow.com/questions/1676384/how-to-pass-flag-to-gcc-in-python-setup-py-script — DavidW, Jul 28 '16 at 20:25
If a cython method is able to return a memory view, then there would be no copying (maybe this [question](http://stackoverflow.com/questions/36357024/wrapping-stdarray-in-cython-and-exposing-it-to-memory-views)) — J.J. Hakala, Jul 28 '16 at 20:31
I searched the contents of `Cython/Includes/libcpp` and there seems to be no `std::move()` defined there (cython 0.24.1). — J.J. Hakala, Jul 28 '16 at 21:48

DavidW · Answer 1 · 2016-07-31T07:52:45.883

A very simple example that duplicates your issue (when compiled incorrectly) and demonstrates how to fix it by changing the compile options.

cdef extern from "<utility>" namespace "std":
    int&& move(int&&)

cdef class someclass:
    cdef int i

    cdef move_from(self, int&& i):
        self.i = move(i)

(Note that I've added namespace "std" when defining move. This is missing in your code, probably because I missed it in the answer you based this code on).

setup.py is as follows

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension('test_move',
              sources=['test_move.pyx'],
              extra_compile_args=['-std=c++11'],
              language='c++')]

If I remove "extra_compile_args" then I see the same errors you report (because the compiler is assuming that you are using the old C++ standard which does not support rvalue references). If I compile it as above then it compiles correctly.

This does not mean there aren't other issues in your code. It is much longer than needed to demonstrate the problem and is not complete (it relies on at least 3 C++ header files that you don't provide). Therefore it is impossible to test.

J.J. Hakala · Answer 2 · 2016-07-28T21:58:04.920

If C++ methods return pointers to objects (or if the ownership of the data can be transferred), and the underlying data can be accessed, memory views should be usable.

The following example is for a vector of integers (int). Class test_view has a reference to the object containing the data and to the view object so the lifetime of those two should be the same.

test.pxd

from libcpp.vector cimport vector

cdef public class test [object cTest, type cTest]:
    cdef vector[int] * test

test.pyx

from libcpp.vector cimport vector

class test_view:
    def __init__(self, test obj):
        cdef ssize_t N = obj.test[0].size()
        cdef int[::1] v = <int[:N]>obj.test[0].data()

        self._obj = obj
        self._view = v

    def get(self):
        return self._view

cdef class test:
    def __cinit__(self):
        self.test = NULL

    def __init__(self, seq):
        self.test = new vector[int]()

        cdef int i

        for i in seq:
            self.test[0].push_back(i)

    def __dealloc__(self):
        print("Dealloc")
        if self.test != NULL:
            del self.test

    # Expose size method of std::vector
    def size(self):
        return self.test[0].size()

    def view(self):
        # return an instance of test_view, object should stay alive
        # for the duration of test_view instance
        return test_view(self)

setup.py

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [ Extension('test', ['test.pyx'], language='c++') ]

setup(name = 'test',
      ext_modules = ext_modules,
      cmdclass = {'build_ext': build_ext})

run.py

import test

a = test.test([1, 2, 3, 4, 5, 6])
v = a.view()

print('Try to cause deallocation')
a = None

print('Print view')
for i in v.get():
    print('\t{:-12d}'.format(i))    

nv = v._view
v = None

print('Not good, should print junk numbers')
for i in nv:
    print('\t{:-12d}'.format(i))

When run.py is executed,

Try to cause deallocation
Print view
                   1
                   2
                   3
                   4
                   5
                   6
Dealloc
Not good, should print junk numbers
            11966656
                   0
                   3
                   4
                   5
                   6

Cython - Exposing C++ (vector and non-vector) objects returned by C++ function to Python

2 Answers2

Linked