Polymorphism and pybind11

Question

I have a strange behaviour with pybind11 when I want to use C++ polymorphism in Python. Here is a simple example of my problem:

import polymorphism as plm

a = plm.mylist()

print(a)

a[0].print()
a[1].print()

The output of this script is

[MyBase, MyDerived]

MyBase

MyBase

but the expected output is

[MyBase, MyDerived]

MyBase

MyDerived

because mylist return a std::vector which contains an instance of a derived class (MyDerived) as a second member. The strange thing is that MyDerived is recognized when I print the list as a whole.

Here is the header file of the C++ code:

/* polymorphism.hpp */

#ifndef POLYMORPHISM_HPP
#define POLYMORPHISM_HPP

#include <vector>

class MyBase
{
  public:
    virtual void print() const;
};


class MyDerived : public MyBase
{
  public:
   virtual void print() const;
};


std::vector<MyBase*> mylist();

#endif

And here is the cpp file:

#include "polymorphism.hpp"

#include <iostream>
#include <pybind11/stl.h>
#include <pybind11/pybind11.h>

void MyBase::print() const
{ std::cout << "MyBase" << std::endl; }

void MyDerived::print() const
{ std::cout << "MyDerived" << std::endl; }

std::vector<MyBase*> mylist()
{
  std::vector<MyBase*> list(2);
  list[0] = new MyBase();
  list[1] = new MyDerived();
  return list;
}

PYBIND11_MODULE(polymorphism, m) 
{
  pybind11::class_<MyBase>(m, "MyBase")
    .def(pybind11::init<>())
    .def("print", &MyBase::print)
    .def("__repr__", [](const MyBase &a) { return "MyBase"; });

  pybind11::class_<MyDerived, MyBase>(m, "MyDerived")
    .def(pybind11::init<>())
    .def("print", &MyDerived::print)
    .def("__repr__", [](const MyDerived &a) { return "MyDerived"; });

  m.def("mylist", &mylist, "return a list");
}

EDIT: more surprisingly, when I remove the binding of "print" for MyDerived, I get the following error message

[MyBase, MyDerived]

MyBase

Traceback (most recent call last):

File "test.py", line 8, in

a[1].print()

AttributeError: 'polymorphism.MyDerived' object has no attribute 'print'

This message seems to mean that MyDerived is well recognized while the wrong version of print is called (if I understand well).

EDIT 2: here is a version using trampoline classes. However, this version leads to the same wrong output.

/* polymorphism.hpp */
#ifndef POLYMORPHISM_HPP
#define POLYMORPHISM_HPP

#include <vector>
#include <pybind11/stl.h>
#include <pybind11/pybind11.h>

class MyBase
{
  public:
    virtual void print() const;
};


class MyDerived : public MyBase
{
  public:
    virtual void print() const;
};


std::vector<MyBase*> mylist();

class PyMyBase : public MyBase 
{
  public:
    using MyBase::MyBase; // Inherit constructors
    void print() const override { PYBIND11_OVERLOAD(void, MyBase, print ); }
};

class PyMyDerived : public MyDerived 
{
  public:
    using MyDerived::MyDerived; // Inherit constructors
    void print() const override { PYBIND11_OVERLOAD(void, MyDerived, print);}
};

#endif

Here is the corresponding cpp file:

/* polymorphism.cpp */
#include "polymorphism.hpp"

#include <iostream>

void MyBase::print() const
{ std::cout << "MyBase" << std::endl; }


void MyDerived::print() const
{ std::cout << "MyDerived" << std::endl; }


std::vector<MyBase*> mylist()
{
  std::vector<MyBase*> list(2);
  list[0] = new MyBase();
  list[1] = new MyDerived();
  return list;
}


PYBIND11_MODULE(polymorphism, m) 
{
   pybind11::class_<MyBase, PyMyBase>(m, "MyBase")
     .def(pybind11::init<>())
     .def("print", &MyBase::print)
     .def("__repr__", [](const MyBase &a) { return "MyBase"; });

   pybind11::class_<MyDerived, PyMyDerived>(m, "MyDerived")
     .def(pybind11::init<>())
     .def("print", &MyDerived::print)
     .def("__repr__", [](const MyDerived &a) { return "MyDerived"; });

   m.def("mylist", &mylist, "return a list");
}

The pybind11 documentation says that you need a so-called trampoline class: http://pybind11.readthedocs.io/en/stable/advanced/classes.html — pschill, Apr 03 '18 at 17:12
Oh yes, I also tried this solution but it does not work. I edit my message in order to give it to you in case you can see what is wrong. — Aleph, Apr 03 '18 at 17:30
Maybe pointers are not well converted in python, so that polymorphism cannot work, as if my vector contains values instead of pointers, i.e. it contains instances of MyBase instead of instances of MyBase*. It would explain why MyDerived is well recognized while the wrong version of print is called. — Aleph, Apr 03 '18 at 18:03
I just played around a little and found that it works if you change the return type of `mylist()` to `vector>`. I dont know why pybind11 has a problem with the raw pointers. — pschill, Apr 03 '18 at 21:16
Thank you! It sounds great! Could you post your solution so that I can see it and declares it as the official solution? — Aleph, Apr 03 '18 at 21:29
I tried your solution and it works fine. Thank you very much. :) — Aleph, Apr 03 '18 at 21:35

score 2 · Accepted Answer · answered Apr 04 '18 at 06:49

I dont know why, but pybind11 seems to have a problem with the raw pointers in mylist(). The example works correctly if you change the return type to vector<unique_ptr<MyBase>>. The following example compiles to a python module example and produces the expected output.

example.cpp:

#include <pybind11/stl.h>
#include <pybind11/pybind11.h>
#include <iostream>
#include <memory>
#include <vector>

class MyBase {
public:
    virtual void print() const {
        std::cout << "MyBase::print()" << std::endl;
    }
};

class MyDerived : public MyBase {
public:
    virtual void print() const override {
        std::cout << "MyDerived::print()" << std::endl;
    }
};

std::vector<std::unique_ptr<MyBase>> mylist() {
    std::vector<std::unique_ptr<MyBase>> v;
    v.push_back(std::make_unique<MyBase>());
    v.push_back(std::make_unique<MyDerived>());
    return v;
}

PYBIND11_MODULE(example, m) {
    pybind11::class_<MyBase>(m, "MyBase")
        .def(pybind11::init<>())
        .def("print", &MyBase::print)
        .def("__repr__", [](MyBase const&) { return "MyBase"; });

    pybind11::class_<MyDerived>(m, "MyDerived")
        .def(pybind11::init<>())
        .def("print", &MyDerived::print)
        .def("__repr__", [](MyDerived const&) { return "MyDerived"; });

    m.def("mylist", &mylist, "returns a list");
}

python shell:

>>> import example
>>> l = example.mylist()
>>> l[0].print()
MyBase::print()
>>> l[1].print()
MyDerived::print()

Polymorphism and pybind11

1 Answers1