How to convert Python object to a std::vector of Cython extension type and back?

Question

I use Cython to wrap C++ code. The code contains a function defined as:

std::vector<ClassOut> analyze(std::vector<ClassIn> inputVec);

ClassIn and ClassOut are extension types. From Python I'd like to be able to call this function with a list or a numpy array (whatever is possible and most sensible). I also want to be able to access and modify the extenion types, so something like this:

run.py

from cythonCode.classIn import PyClassIn
from cythonCode.classOut import PyClassOut
from cythonCode.analyze import PyAnalyze

classIn_list = []
classIn_list.append(PyClassIn())
classIn_list.append(PyClassIn())
classOut_list = PyAnalyze(classIn_list)
print(classOut_list)

The wrappers PyClassIn and PyClassOut work fine. The problem is simply the wrapping of the analyze function from the beginning. My version of the wrapper PyAnalyze can be found below:

analyze.pxd

from libcpp.vector cimport vector
from classOut cimport ClassOut
from classIn cimport ClassIn, PyClassIn

cdef extern from "../cppCode/analyze.h":
  vector[ClassOut] analyze(vector[ClassIn])

analyze.pyx

def PyAnalyze(vector<PyClassIn> inputVec)
  return analyze(inputVec)

There are for sure mistakes in the analyze.pyx. I am getting the error:

Python object type 'PyClassIn' cannot be used as a template argument

The return statement has to be incorrect as well. Cython complains with:

Cannot convert 'vector[ClassOut]' to Python object

I have this code as a minimal example at https://github.com/zyzzler/cython-vector-minimal-example.git

EDIT: Thanks to your input I am now at the point where the return type of the definition can be wrapped but the argument not yet. The link in the first comment provided great information about getting the return type correct. So assuming I'd want to wrap a function defined as:

std::vector<ClassOut> analyze(std::vector<float> inputVec);

everything works fine! However, I have to deal with the Extension Type ClassIn in place of float. So below is the code I have now:

analyze.pyx

def PyAnalyze(classesIn):
  cdef vector[ClassOut] classesOut = analyze(classesIn)

  retval = PyClassOutVector()
  retval.move_from(move(classesOut))

  return retval

The above code throws the error:

Cannot convert Python object to 'vector[ClassIn]'

The reason for this error is clear. "classesIn" is a Python list of PyClassIn objects but analyze(...) takes a vector[ClassIn] as input. So the question is how to convert from the Python list to the std::vector and/or from PyClassIn to ClassIn? I tried to use the rvalue reference and move constructor formalism as well but it didn't work. I also tried to do it via a function like this:

cdef vector[ClassIn] list_to_vec(classInList):
  cdef vector[ClassIn] classInVec
  for classIn in classInList:
    classInVec.push_back(<ClassIn>classIn)
  return classInVec

The problem here is the <ClassIn>classIn statement. It says:

no matching function for call to 'ClassIn::ClassIn(PyObject*&)'

So I am really puzzled here. How could this be solved? I adapted the code with the minimal example in the git I posted above.

EDIT2: To provide some more information for the comments below. I now have a wrapper for PyClassInVector exactly like the one for PyClassOutVector, see below:

cdef class PyClassInVector:
  cdef vector[ClassIn] vec

  cdef move_from(self, vector[ClassIn]&& move_this):
    self.vec = move(move_this)

  def __getitem__(self, idx):
    return PyClassIn2(self, idx)

  def __len__(self):
    return self.vec.size()

cdef class PyClassIn2:
  cdef ClassIn* thisptr
  cdef PyClassInVector vector

  def __cinit__(self, PyClassInVector vec, idx):
    self.vector = vec
    self.thisptr = &vec.vec[idx]

In analyze.pxd I also added:

cdef extern from "<utility>":
  vector[ClassIn]&& move(vector[ClassIn]&&)

Now based on the comments, in the PyAnalyzefunction I'd do:

def PyAnalyze(classesIn):
  # classesIn is a list of PyClassIn objects and needs to be converted to a PyClassInVector
  classInVec = PyClassInVector()
  cdef vector[ClassOut] classesOut = analyze(classInVec.vec)

  retval = PyClassOutVector()
  retval.move_from(move(classesOut))

  return retval

But as the comment in the code says, how can I get the list of PyClassIn objects (classesIn) into the PyClassInVector (classInVec)?

EDIT3: Imagine PyClassOut is decorated with an attribute that can be set via the constructor:

cdef class PyClassOut()
  def __cinit__(self, number):
    self.classOut_c = ClassOut(number)

  @property
  def number(self):
    return self.classOut_c.number

In run.py I'm doing something like this:

from cythonCode.classIn import PyClassIn
from cythonCode.classOut import PyClassOut
from cythonCode.analyze import PyAnalyze

classIn_list = []
classIn_list.append(PyClassIn(1))
classIn_list.append(PyClassIn(2))

classOut_list = PyAnalyze(classIn_list)

print(classOut_list[0].number)
print(classOut_list[1].number)

classOut_list is essentially the retvalue from the PyAnalyze function. The retvalue is a PyClassOutVector object. So classOut_list[0] gives me the PyClassOut2 object at the index 0. But here I don't have access to the attribute number. Also what I notice is that the address of classOut_list[1] is the same as the one of classOut_list[0]. I don't understand this. I am not entirely sure what 'move' does. Also I actually want to have a python list again as the retvalue, ideally with PyClassOut objects instead of the PyClassOut2 objects. Does that make sense? And is it feasible?

Something based on [this answer](https://stackoverflow.com/a/33679481/4657412) might be suitable - it lets you wrap the vectors in Cython. — DavidW, Sep 30 '19 at 19:49
Thanks a lot for your input! This indeed helped to fix one part of my issue. I made an EDIT explaining it. For the other part I am still puzzled. — zyzzler, Oct 01 '19 at 11:39
classIn is a `PyClassIn` object - to use your scheme you'd have to tell Cython that, and then get the object it wraps: `classInVec.push_back(classIn.thisptr)` (or something similar). You could also consider defining a `ClassInVector` Python wrapper - that'd let you pass to `analyse` without a copy if you changed its argument to a const reference. — DavidW, Oct 01 '19 at 12:10
(I've updated my answer on the linked question slightly - on re-reading my example for `PyClassIn` goes wrong if the vector is resized. Unless you're sure that won't happen then it might be worth looking at the update) — DavidW, Oct 01 '19 at 12:11
But can your example from the linked question be applied to PyClassIn? I tried that by defining a ```PyClassInVector``` with a ```move_from(...)``` included. But it has to be the other way around compared to ```PyClassOutVector```. I couldn't figure out how to code this. So in relation to the const reference in the analyze argument you mentioned, how whould I do this? — zyzzler, Oct 01 '19 at 12:35
You don't really need to move anything in that case, just pass `classInVec.thisptr` to `analyze` (and declare the type of `classInVec` to be `PyClassInVector`) — DavidW, Oct 01 '19 at 12:43
Sorry, I don't understand it. I needed to make an EDIT2 to clearify my standpoint. Hope this is understandable and easy to solve. — zyzzler, Oct 01 '19 at 13:20
Thinking about the approach with the `list_to_vec` function, I'd need to expand the `PyClassInVector` by a special method `__setitem__`. Otherweise I don't see how it will be possible to transfer the list of PyClassIn objects into the vector of ClassIn objects, do you? — zyzzler, Oct 01 '19 at 14:47
I envisaged that you'd just create a `PyClassInVector` rather than a `list`. However, it sounds like you might be better just working from a list like you were trying to do in edit 1 - that code looked pretty close to working — DavidW, Oct 01 '19 at 14:56
The `list_to_vec` function is working now. But I have problems with the `PyClassOutVector`. I went over the changes you did to your original post in the linked question to see if that helps. I noticed one thing while implementing your changes: in `__cinit__` it should be `self.idx = idx` instead of the `self.thisptr...` stuff. However, my problem remains. I describe it in EDIT3. — zyzzler, Oct 02 '19 at 13:55
If you'd rather have a list as the input and output then there's no reason to use `PyClassOutVector` and `PyClassInVector`. I also suggest you forget about `move` and just copy things element-by-element in `PyAnalyze`. It was an approach that I prefer (mostly to avoid copying) but I think it's a bit confusing and isn't helping you. — DavidW, Oct 02 '19 at 15:37

DavidW · Accepted Answer · 2019-10-07T13:18:20.707

In the comments I tried to recommend a solution involving wrapping C++ vectors. I prefer this approach because it avoids copying memory multiple times, but I think it's causing more confusion and you'd rather just use Python lists. Sorry.

To use Python lists you just have to copy the input and output within PyAnalyze. You have to do it manually - no automatic conversions exists. You also have to be aware of the difference between your wrapped classes and the underlying C++ classes. You can only send the C++ classes to C++, not the wrapped ones.

Dealing with the input is easy:

 def PyAnalyze(classesIn):
     # classesIn is a list of PyClassIn objects and needs to be converted to a PyClassInVector
     cdef vector[ClassIn] vecIn
     cdef vector[ClassOut] vecOut
     cdef PyClassIn a
     for a in classesIn:
         # need to type a to access its C attributes
         # Cython should check that a is of the correct type
         vecIn.push_back(a.classIn_c)

     vecOut = analyze(vecIn)

Returning the data back to Cython wrapped as PyClassOut is a little more difficult since you can't send a C++ type to a Cython constructor (all arguments to constructors must be Python types). Just construct an empty PyClassOut then copy the new data into it. Again, work through your vector element by element

 def PyAnalyze(classesIn):
     cdef PyClassOut out_val
     # ... use code above ...
     out_list = []
     for i in range(vecOut.size()):
        out_val = PyClassOut()
        out_val.classOut_c = vecOut[i]
        out_list.append(out_val)
     return out_list

Thank you very much, this was what I have been looking for! However, the r-reference and move-constructor way might still be interesting because it is more high-performance. Do you see a way how to implement this within the minimal example? — zyzzler, Oct 07 '19 at 13:24
Reference is easy: simply change the signature of `analyze` to `std::vector analyze(const std::vector& inputVec)`. This is the standard C++ way of passing around large objects that you don't want to change. You shouldn't need to modify the Cython code — DavidW, Oct 07 '19 at 13:57
Move constructor is harder, and I probably wouldn't do it unless you understand the C++ side of it well. To apply it to the output: `out_val.classOut_c = move(vecOut[i])`. You need to create a "move assignment operator" for `ClassOut` and a Cython wrapping of `std::move` (probably just for `ClassOut` and not a template function). I don't think you can use `move` on the input since you still have the objects on the Python/Cython side. — DavidW, Oct 07 '19 at 14:01

How to convert Python object to a std::vector of Cython extension type and back?

1 Answers1