pybind11: Python to C++ Datatype Conversion Not Working

Question

Problem

I am trying to convert a list of lists, returned by a python function called inside a C++ code. Though the pybind11 library allows type conversion from python data types to C++ data types, my attempt to convert a list of lists returned by python to a std::list of std::list of strings of C++, fails every time.

Code

Here is the python function (The function returns a list of list containing string values):

def return_sheet(self):

     """Returns the sheet in a list of lists

     """

     dataTable = []

     for r in range(self._isheet.nrows):

         datalist = []

         for c in range(self._isheet.ncols):

             datalist.append(self._isheet.cell_value(r,c))

         dataTable.append(datalist)

 return dataTable

And Here I am calling it in C++ using pybind11:

py::list obj = _tool.attr("return_sheet")();

data = py::cast<SheetData>(obj); // This is where the problem lies, This cast crashes the program

Where SheetData is a typedef for:

typedef std::list<std::list<std::string> > SheetData;

While debugging, I found out that the program is actually crashing at this line:

py::object dataTable = _tool.attr("return_sheet")(); // Where _tool.attr("return_sheet")() gives an py::object which is a list of list of str

Does someone know, how can I successfully convert a list of lists of python to std::list of std::list of C++?

EDIT

Here is the python program file I am embedding in c++ [xlanalyser.py] : https://pastebin.com/gARnkMTv

And Here is the c++ code [main.cpp] : https://pastebin.com/wDDUB1s4

Note: All other functions in xlanalyser.py do not cause a crash on embedding in c++ [ only the return_sheet() function causes the crash ]

CarlosAlbaladejo · Accepted Answer · 2019-03-24T01:22:33.320

0

You can use the Python/C API as a workaround (Check function CastToSheetData). I include the full example below:

program.py

def return_matrix():

    dataTable = []
    for r in range(0,2):
        datalist = []

        for c in range(0,2):
            datalist.append(str(r+c))

        dataTable.append(datalist)
    return dataTable

main.cpp

#include <pybind11/embed.h>
#include <iostream>
#include <list>
#include <string>

typedef std::list<std::list<std::string> > SheetData;

namespace py = pybind11;

SheetData CastToSheetData(PyObject *obj)
{
    SheetData data;
    PyObject *iter = PyObject_GetIter(obj);

    if (!iter)
        return data;
    while (true) {
        std::list<std::string> aux_list;
        PyObject *next = PyIter_Next(iter);
        if (!next) {
            // nothing left in the iterator
            break;
        }
        PyObject *iter2 = PyObject_GetIter(next);
        if (!iter2)
            continue;
        while(true) {
            PyObject *next2 = PyIter_Next(iter2);
            if (!next2) {
                // nothing left in the iterator
                break;
            }
            PyObject* pyStrObj = PyUnicode_AsUTF8String(next2); 
            char* zStr = PyBytes_AsString(pyStrObj); 
            std::string foo(strdup(zStr));
            aux_list.push_back(foo);
            Py_DECREF(pyStrObj);
        }
        data.push_back(aux_list);
    }

    return data;
}


int main()
{
    py::scoped_interpreter guard{};
    py::module calc = py::module::import("program");
    py::object result = calc.attr("return_matrix")();

    SheetData data = CastToSheetData(result.ptr());

    for (auto l : data)
    {
        std::cout << "[ ";
        for(auto s : l)
            std::cout << s <<  " ";
        std::cout << "]" << std::endl;
    }

    return 0;
}

Output:

[ 0 1 ]
[ 1 2 ]

Probably, the best way to go here is to make a custom type_caster using code like the one in the CastToSheetData function inside the load method

edited Mar 24 '19 at 01:22

answered Mar 24 '19 at 01:15

CarlosAlbaladejo

90
2
8

Thanks for helping me out but the solution provided doesn't works. Your code crashes at the line of importing module [ py::module calc = py::module::import("program"); ] in main function. Whereas in my code the program crahes at the line of getting data from python function 'return sheet' [ py::object dataTable = _tool.attr("return_sheet")(); ] . Can you please help figure it out? – MAXEINSTEIN Mar 26 '19 at 09:49
It didn't crash for me. It looks like your problem is not related with the datatype conversion but with embedding your python function into C++. Could you post your complete code to check how you import the module `_tool`? – CarlosAlbaladejo Mar 26 '19 at 14:42
And did you add my python code into a **program.py** file in the same folder as the **main.cpp**? – CarlosAlbaladejo Mar 26 '19 at 14:44
Of course I did add your python program to the correct location.. I m using Qt 5.12.1 with MSVC 32 bit for the project. I am going to post my c++ and python code to the end of my question description. Thanks for taking out your time helping me. – MAXEINSTEIN Mar 26 '19 at 18:59
Your code works fine on my computer. I printed an excel without problem. I am using 64 bits (I compiled pybind11 for 64 bits). More detailed, I use VSCode with CMake 2.8, MSVC 19.12.25835.0 64 bits, my python version is 3.7 and pybind11 v2.3dev0. And don't worry, I enjoy doing this haha – CarlosAlbaladejo Mar 27 '19 at 19:02
It also crashes for me when the path or the filename of the excel sheet is not correct – CarlosAlbaladejo Mar 27 '19 at 19:05
@MAXEINSTEIN maybe you should check the answer as solution as long as it solves your question, though finally your problem was somewhere else. It could help other people with same issue – CarlosAlbaladejo Apr 11 '19 at 15:28
Alright Carlos , I will mark it answered, but the thing is that maybe it's the problem with pybind11 because the same thing works fine with Python C API for me. Maybe pybind11 can't handle deeply nested data structures. Once again, thanks for all the time you helped me with. – MAXEINSTEIN Apr 18 '19 at 16:29

pybind11: Python to C++ Datatype Conversion Not Working

Problem

Code

EDIT

1 Answers1