Cython with Python3.5: convert list of string to char**

Question

I am trying to pass a list of string to a C++ function that takes a char** as argument using Cython.

I tried other solutions I can't remember but I mainly tried the two following options:

Using a convertion function extracted from here, which is basically the same answer than here, but it raises an error, on the g++ compilation which is

cy_wrapper.cpp: In function 'char** _pyx_f_10cy_wrapper_to_cstring_array(PyObject*)':
cy_wrapper.cpp:1223:44: error 'PyString_AsString' was not declared in this scope
    __pyx_t_5 = PyString_AsString(__pyx_t_4); if (unlikely(__pyx_t_5 == ((char *)NULL))

Using the encoding method of python strings, as suggested in here but the answer is not satisfying since it raises the error:
```
cy_wrapper.pyx:14:35: Storing unsafe C derivative of temporary Python reference
```

I looked in the library files at ~/.local/lib/python3.5/site-packages/Cython/Includes/cpython and I found the file string.pxd which contains the function PyString_AsString I need.

Why is it not found? If there is no possibility to use it, is there a workaround?

I am working on Ubuntu 16.04.4 LTS (tegra kernel) on a arm64 arch.

My cy_wrapper.pyx is exactly like this:

from cpython.string cimport PyString_AsString
from libc.stdlib cimport malloc

cdef extern:
    cdef cppclass imageNet:
        imageNet* Create(int argc, char** argv)

cdef char** to_cstring_array(list_str):
    cdef char** ret = <char **>malloc(len(list_str) * sizeof(char *))
    for i in range(len(list_str)):
        ret[i] = PyString_AsString(list_str[i])
    return ret

cdef class PyImageNet:
    cdef imageNet* c_net
    def Create(self, argc, kwargs):
        cdef char** c_argv = to_cstring_array(kwargs)
        return PyImageNetFactory(self.c_net.Create(argc, c_argv))

cdef object PyImageNetFactory(imageNet* ptr):
    cdef PyImageNet py_obj = PyImageNet()
    py_obj.c_net = ptr
    return py_obj

My setup.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
      cmdclass = {"build_ext": build_ext},
      ext_modules = [
          Extension("cy_wrapper",
                    sources=["cy_wrapper.pyx"],
                    libraries=["shared_inference"],
                    language="c++",
                    extra_compile_args=["-O3", "-Wall"],
                    extra_link_args=["-L../build/"]
                    )
          ]
)

Ofc libshared_inference.so is located in ../build/ and defines the class imageNet.

The problem is that `PyString_AsString` is a python2 functionality, which is not present in python3. Cython-documentation covers this topic here: http://cython.readthedocs.io/en/latest/src/tutorial/strings.html#accepting-strings-from-python-code — ead, Mar 16 '18 at 12:28
There's also a memory management disaster hidden in here (as "Storing unsafe C derivative of temporary Python reference" is warning you). In both cases the memory is owned by the Python string, so the memory is only valid as long as the Python object is valid. — DavidW, Mar 16 '18 at 13:20

score 1 · Accepted Answer · answered Mar 20 '18 at 13:44

For those interested, here is how I converted my list of string into a char** (I forgot to mention in my question that my method was static, but it does not really matter regarding my solution), hope this will help someone.

# cy_wrapper.pyx
from libc.stdlib cimport malloc, free

cdef extern:
    cdef cppclass imageNet:
        @staticmethod
        imageNet* Create(int argc, char** argv)

cdef class PyImageNet:
    cdef imageNet* c_net
    @staticmethod
    def Create(args):
        # Declare char**
        cdef char** c_argv
        # Allocate memory
        c_argv = <char**>malloc(len(args) * sizeof(char*))
        # Check if allocation went fine
        if c_argv is NULL:
            raise MemoryError()
        # Convert str to char* and store it into our char**
        for i in range(len(args)):
            args[i] = args[i].encode()
            c_argv[i] = args[i]
        # Grabbing return value
        cdef imageNet* c_tmp_net = imageNet.Create(len(args), c_argv)
        # Let him go
        free(c_argv)
        # Return python-compatible value
        return PyImageNetFactory(c_tmp_net)

cdef object PyImageNetFactory(imageNet* ptr):
    cdef PyImageNet py_obj = PyImageNet()
    py_obj.c_net = ptr
    return py_obj

Test code:

# test.py
import cy_wrapper
args = ["str1", "str2"]
net = cy_wrapper.PyImageNet.Create(args)

Cython with Python3.5: convert list of string to char**

1 Answers1