Boost Python Exposing C++ class with constructor taking a std::list

Question

I have a class that looks like the following,

class MyClass
{
    MyClass( std::list<std::string> );

};

I tried exporting it to python using,

class_<MyClass, boost::noncopyable >("MyClass", init<std::list<std::string>>());

But i got a signature mismatch error,

did not match C++ signature:
__init__(_object*, std::__cxx11::list<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > )

Can anyone advise how to go about it?

I presume that you want to be able to construct this object in Python by passing it a Python list? — Dan Mašek, May 24 '19 at 14:51
Furthermore, do I correctly assume that you want `MyClass` to still use `std::list` internally (i.e. the wrapping should be non-intrusive)? — Dan Mašek, May 24 '19 at 15:18
@DanMašek That's my thought too and I started looking at `boost::python::extract` etc. but I mostly found manual transfers between boost's python list and standard C++ containers where I'd expect it to accept iterators etc. If you've got any hints of how to improve my answer please leave a comment and I'll do my best. — Ted Lyngmo, May 24 '19 at 15:55
@TedLyngmo I've got [two possible approaches](https://pastebin.com/0YgUVbkH), but I need to polish it a bit more and write up an explanation once the OP confirms my assumptions. Simpler (if we just care about constructor) is to provide a custom standalone function which converts the list and constructs `MyClass` afterwards (and bind it as `__init__`). Second, more complex but more flexible is to provide to/from python converters for the list of strings. — Dan Mašek, May 24 '19 at 16:16
@DanMašek Great, you go ahead and polish that up and post an answer so I can remove mine (even though it does "work") :-) — Ted Lyngmo, May 24 '19 at 16:21
@TedLyngmo Posted.. that's the best I can come up with tonight. Feel free to review it and point out anything that could be explained better. I'll have another look at it tomorrow, there are few things that could be done better, but I'm getting tired. — Dan Mašek, May 25 '19 at 00:37
@DanMašek Nice! +1 from me! The only disappointment I feel is the lack of integration support from these two major contributing communities. Boost is a bit of a disappointment in this regard - Forefront in some cases, but having to do what you so nicely did for seamless integration makes me feel like they have left the art of simplifying things behind. Nicely done on your part! Danny: You may want to move the accepted answer over to Dans now. It's much more thorough and I'll happily remove mine to not lead people astray later. — Ted Lyngmo, May 25 '19 at 02:22

Dan Mašek · Accepted Answer · 2019-05-25T00:38:54.333

Two possible approaches come to mind.

Let's assume we're trying to expose the following C++ class, non-intrusively:

class MyClass
{
public:
    MyClass(std::list<std::string> messages)
        : msgs(std::move(messages))
    {
    }

    void dump() const
    {
        std::cout << "Messages:\n";
        for (auto const& msg : msgs) {
            std::cout << msg << "\n";
        }
    }

    // NB: This would be better returned as const ref
    //     but I have issues exposing that to Python
    //     that I have yet to solve
    std::list<std::string> messages() const
    {
        return msgs;
    }

private:
    std::list<std::string> msgs;
};

If the only place we need to deal with std::list is the constructor, then the simplest approach is to write a small "factory" function which will

Take a Python list as an input.
Create a std::list and populate it with values from the Python list.
Instantiate MyClass with the std::list.
Return that instance of MyClass.

We'll use a shared_ptr to deal with the memory management. To easily initialize the std::list, we can take advantage of boost::python::stl_input_iterator.

boost::shared_ptr<MyClass> create_MyClass(bp::list const& l)
{
    std::list<std::string> messages{ bp::stl_input_iterator<std::string>(l)
        , bp::stl_input_iterator<std::string>() };
    return boost::make_shared<MyClass>(messages);
}

Once we have this function, we'll expose it in place of the original MyClass constructor. To do this, we first need to disable any default constructor bindings, so we use boost::python::no_init. In python, constructors are simply functions named __init__. Finally, we need to use the apparently undocumented function boost::python::make_constructor to create an appriate function object.

BOOST_PYTHON_MODULE(so07)
{
    bp::class_<MyClass, boost::noncopyable, boost::shared_ptr<MyClass>>("MyClass", bp::no_init)
        .def("__init__", bp::make_constructor(create_MyClass))
        .def("dump", &MyClass::dump)
        ;
}

Transcript:

>>> import so07
>>> test = so07.MyClass(['a','b','c'])
>>> test.dump()
Messages:
a
b
c

If we wish to use std::list in other contexts, then writing individual wrapper functions to deal with the translation would quickly get out of hand. To avoid this, we can register custom converters that will allow Boost.Python to automatically convert Python lists holding strings to std::list<std::string> objects and vice versa.

Going from C++ to python is quite simple -- just construct a boost::python::list and then add all the elements from the C++ list. We can register this converter using boost::python::to_python_converter.

struct std_list_to_python
{
    static PyObject* convert(std::list<std::string> const& l)
    {
        bp::list result;
        for (auto const& value : l) {
            result.append(value);
        }
        return bp::incref(result.ptr());
    }
};

Going from Python to C++ is a two-step process. First of all, we need a function to determined whether the input is a valid candidate for conversion. In this case, the object needs to be a Python list, and each of its elements needs to be a Python string. The second steps consists of in-place construction of the std::list and subsequent population of it with the elements from the Python list. We register this converter using boost::python::converter::registry::push_back.

struct pylist_converter
{
    static void* convertible(PyObject* object)
    {
        if (!PyList_Check(object)) {
            return nullptr;
        }

        int sz = PySequence_Size(object);
        for (int i = 0; i < sz; ++i) {
            if (!PyString_Check(PyList_GetItem(object, i))) {
                return nullptr;
            }
        }

        return object;
    }

    static void construct(PyObject* object, bp::converter::rvalue_from_python_stage1_data* data)
    {
        typedef bp::converter::rvalue_from_python_storage<std::list<std::string>> storage_type;
        void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;

        data->convertible = new (storage) std::list<std::string>();

        std::list<std::string>* l = (std::list<std::string>*)(storage);

        int sz = PySequence_Size(object);
        for (int i = 0; i < sz; ++i) {
            l->push_back(bp::extract<std::string>(PyList_GetItem(object, i)));
        }
    }
};

Our module will look as follows:

BOOST_PYTHON_MODULE(so07)
{
    bp::to_python_converter<std::list<std::string>, std_list_to_python>();

    bp::converter::registry::push_back(&pylist_converter::convertible
        , &pylist_converter::construct
        , bp::type_id<std::list<std::string>>());

    bp::class_<MyClass, boost::noncopyable>("MyClass", bp::init<std::list<std::string>>())
        .def("dump", &MyClass::dump)
        .def("messages", &MyClass::messages)
        ;
}

Transcript:

>>> import so07
>>> test = so07.MyClass(['a','b','c'])
>>> test.dump()
Messages:
a
b
c
>>> test.messages()
['a', 'b', 'c']

References:

Complete Code:

#include <boost/python.hpp>
#include <boost/python/stl_iterator.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/make_shared.hpp>

#include <list>
#include <iostream>

namespace bp = boost::python;

class MyClass
{
public:
    MyClass(std::list<std::string> messages)
        : msgs(std::move(messages))
    {
    }

    void dump() const
    {
        std::cout << "Messages:\n";
        for (auto const& msg : msgs) {
            std::cout << msg << "\n";
        }
    }

    std::list<std::string> messages() const
    {
        return msgs;
    }

private:
    std::list<std::string> msgs;
};

#if 1

boost::shared_ptr<MyClass> create_MyClass(bp::list const& l)
{
    std::list<std::string> messages{ bp::stl_input_iterator<std::string>(l)
        , bp::stl_input_iterator<std::string>() };
    return boost::make_shared<MyClass>(messages);
}

BOOST_PYTHON_MODULE(so07)
{
    bp::class_<MyClass, boost::noncopyable, boost::shared_ptr<MyClass>>("MyClass", bp::no_init)
        .def("__init__", bp::make_constructor(create_MyClass))
        .def("dump", &MyClass::dump)
        ;
}

#else

struct std_list_to_python
{
    static PyObject* convert(std::list<std::string> const& l)
    {
        bp::list result;
        for (auto const& value : l) {
            result.append(value);
        }
        return bp::incref(result.ptr());
    }
};


struct pylist_converter
{
    static void* convertible(PyObject* object)
    {
        if (!PyList_Check(object)) {
            return nullptr;
        }

        int sz = PySequence_Size(object);
        for (int i = 0; i < sz; ++i) {
            if (!PyString_Check(PyList_GetItem(object, i))) {
                return nullptr;
            }
        }

        return object;
    }

    static void construct(PyObject* object, bp::converter::rvalue_from_python_stage1_data* data)
    {
        typedef bp::converter::rvalue_from_python_storage<std::list<std::string>> storage_type;
        void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;

        data->convertible = new (storage) std::list<std::string>();

        std::list<std::string>* l = (std::list<std::string>*)(storage);

        int sz = PySequence_Size(object);
        for (int i = 0; i < sz; ++i) {
            l->push_back(bp::extract<std::string>(PyList_GetItem(object, i)));
        }
    }
};


BOOST_PYTHON_MODULE(so07)
{
    bp::to_python_converter<std::list<std::string>, std_list_to_python>();

    bp::converter::registry::push_back(&pylist_converter::convertible
        , &pylist_converter::construct
        , bp::type_id<std::list<std::string>>());

    bp::class_<MyClass, boost::noncopyable>("MyClass", bp::init<std::list<std::string>>())
        .def("dump", &MyClass::dump)
        .def("messages", &MyClass::messages)
        ;
}

#endif

Thank you for the very comprehensive answer! – Danny May 27 '19 at 03:26 — Danny, May 27 '19 at 03:26

Ted Lyngmo · Answer 2 · 2019-05-24T14:58:12.103

0

It's my first go at this but it turns out boost has their own python list type

This actually does work:

#include <boost/python.hpp>
#include <boost/python/list.hpp>
#include <boost/python/extract.hpp>
#include <string>

using namespace boost::python;

struct MyClass {
    MyClass(boost::python::list messages) : msgs(messages) {}
    void set(boost::python::list messages) { msgs = messages; }
    boost::python::list get() { return msgs; }

    boost::python::list msgs;
};

BOOST_PYTHON_MODULE(my_first) {
    class_<MyClass, boost::noncopyable>("MyClass", init<boost::python::list>())
        .def("get", &MyClass::get)
        .def("set", &MyClass::set);
}

A session:

PYTHONPATH="." python3
>>> from my_first import MyClass
>>> a = MyClass(['a', 'b'])
>>> b = a.get()
>>> print(b)
['a', 'b']

edited May 24 '19 at 14:58

answered May 24 '19 at 11:53

Ted Lyngmo

93,841
5
60
108

oh wait, sorry. did you try calling it in python? – Danny May 24 '19 at 12:35
like MyClass(['abc', 'def']) in python – Danny May 24 '19 at 12:36
ahh, the error i got was when trying to call it from python. – Danny May 24 '19 at 12:41
i don't use bjam, i'm using cmake mostly, since i'm on the CLion environment – Danny May 24 '19 at 12:41
if do try the code you will get the error msg that i mentioned above. – Danny May 24 '19 at 13:54

Boost Python Exposing C++ class with constructor taking a std::list

2 Answers2