Cannot iterate over a std::vector wrapped with SWIG when using custom typemaps

Question

I'm wrapping a class using SWIG typemaps, and a std::vector of this class using "std_vector.i" provided by SWIG. The target language is Python. Everything seems to work except that I cannot iterate over the vector.

I've created a minimal example about complex numbers (for convenience only, it has nothing to do with my real project).

This is the C++ class I'd like to wrap:

struct Komplex
{
   Komplex() : real(0.0), imag(0.0) {}
   double real;
   double imag;
};

This is my SWIG interface file:

%module myproject
%{ 
#include "komplex.h"
%}

%typemap(typecheck) Komplex& 
{ 
    PyComplex_Check($input) ? 1 : 0; 
}

%typemap(in) Komplex& (Komplex temp) 
{ 
    temp.real = PyComplex_RealAsDouble($input);
    temp.imag = PyComplex_ImagAsDouble($input);
    $1 = &temp; 
}

%typemap(out) Komplex& 
{
    $result = PyComplex_FromDoubles($1->real, $1->imag);
}

// define a vector of the wrapped class:
%include "std_vector.i"
%template(KomplexVector) std::vector<Komplex>;

I can simply test the typecheck/in/out typemaps using the following Python code:

import myproject

# fill a vector (the "typecheck" and "in" typemaps are used!)
vec = myproject.KomplexVector()
vec.append( complex(1,2) )

# below, the output typemap is used:
print("First attempt:")
for i in xrange(len(vec)):
    print(vec[i])                 # prints: (1+2j)
    assert vec[i] == complex(1,2) # OK

# below, the output typemap is NOT used:
print("Second attempt:")
for k in vec:
    print(k)                 # prints: <Swig Object of type 'Komplex *' at 0x7f0194c6de10>
    assert k == complex(1,2) # fails!

As you can see in the output, iterating over the vector results in an opaque value:

First attempt:
(1+2j)
Second attempt:
<Swig Object of type 'Komplex *' at 0x7f0194c6de10>
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    assert k == complex(1,2) # fails!
AssertionError
swig/python detected a memory leak of type 'Komplex *', no destructor found.

So my question is simple: how can I get it to work?

I've tried the following:

• brute force, i.e. defining all possibly relevant typemaps I can imagine:
  • more in/out/varout/... typemaps for Komplex values/pointers/references/const references/...
  • defining in/out/varout/... typemaps for std::vector<Komplex>::value_type values/pointers/references/..
  • same for std::vector<Komplex>::difference_type / iterator / const_iterator / ...
• using the latest SWIG version from github
• copying the approach from here
• a bunch of other things, like changing the order of the contents in the interface file

... all to no avail.

The full source code (including a little build script for Linux/g++) can be found here.

Any help appreciated!

---

Edit: There are a few workarounds that avoid the lack of a properly wrapped iterator, but none of them are pretty (I think). E.g. one possibility is to define a custom Iterator object in Python, and let SWIG modify the iterator and insert methods in the Python code.

For instance, the following code will make sure that for k in vec: ... and vec.insert(1, complex(4,5)) will produce the expected behavior:

%pythoncode %{
class MyVectorIterator(object):

    def __init__(self, pointerToVector):
        self.pointerToVector = pointerToVector
        self.index = -1

    def next(self):
        self.index += 1
        if self.index < len(self.pointerToVector):
            return self.pointerToVector[self.index]
        else:
            raise StopIteration
%}

%rename(__cpp_iterator) std::vector<Komplex>::iterator;
%rename(__cpp_insert) std::vector<Komplex>::insert;

%extend std::vector<Komplex> {
%pythoncode {
    def iterator(self):
        return MyVectorIterator(self)
    def insert(self, i, x):
        if isinstance(i, int): # "insert" is used as if the vector is a Python list
            _myproject.KomplexVector___cpp_insert(self, self.begin() + i, x)
        else: # "insert" is used as if the vector is a native C++ container
            return _myproject.KomplexVector___cpp_insert(self, i, x)
   }
}

%include "std_vector.i" 
%template(KomplexVector) std::vector<Komplex>;

I wouldn't call it a "solution" (let alone a "nice" solution) but it can serve as a temporary workaround I guess...

Answer 1

It's a bit late, but let me record the solution to a similar problem I found here: things seem to work just fine if you simply define SWIG_NO_EXPORT_ITERATOR_METHODS before including any SWIG headers. This is a rather blunt tool because it completely disables iterator support, just as it says. It still allows things to work, and work optimally, for vectors however as Python falls back to calling __getitem__ in a loop until it fails if __iter__ is not implemented by the object and this is exactly how iterating over vectors works anyhow.

Notice that SWIG also generates __getitem__ for other container types, e.g. std::list and std::set, but its implementation is, of course, inefficient for them, so this workaround is not without its costs if you use these containers.

Answer 2

I was able to get rid of the __iter__ functions in automatically generated wrappers for STL containers by adding the following line (I found the symbol in share/swig/python/pyiterators.swg) at the top of my .i file

#define SWIGPYTHON_BUILTIN

After this, the generated .py file did not contain __iter__ routines anymore, so Python falls back to using __getitem__ as described in other answers above. Don't know why this worked. (SWIG is just trial&error 75% of the time anyway -_-)

But this might be quite a hack, because I'm assuming that this define is normally only set if SWIG flag -builtin is used. But I didn't do that, I merely added the define. So this might cause undefined behaviour. Yet, for me it works so far...