How to convert numpy arrays to vector& (reference) with SWIG

Question

My goal:

Create 3 numpy arrays in python (2 of them will be initialized with specific values), then send all three of them through swig into a c++ function as vector references (this is in order to avoid copying over the data and loosing efficiency). Once in the c++ function, add 2 of the arrays and put their sum in the 3rd array.

vec_ref.h

#include <vector>
#include <iostream>

void add_vec_ref(std::vector<int>& dst, std::vector<int>& src1, std::vector<int>& src2);

vec_ref.cpp

#include "vec_ref.h"
#include <cstring> // need for size_t
#include <cassert>

void add_vec_ref(std::vector<int>& dst, std::vector<int>& src1, std::vector<int>& src2) {
    std::cout << "inside add_vec_ref" << std::endl;
    assert(src1.size() == src2.size());
    dst.resize(src1.size());

    for (size_t i = 0; i < src1.size(); i++) {
        dst[i] = src1[i] + src2[i];
    }
}

vec_ref.i

%module vec_ref
%{
    #define SWIG_FILE_WITH_INIT
    #include "vec_ref.h"
%}

%include "numpy.i"
%init %{
import_array();
%}

%include "std_vector.i"
%template(vecInt) std::vector<int>;
// %template(vecIntRef) std::vector<int> &; 

// %apply (std::vector<int> * INPLACE_ARRAY1, int DIM1) {(std::vector<int> * dst, int a),(std::vector<int> * src1, int b),(std::vector<int> * src2, int c)};
// %apply (std::vector<int> * INPLACE_ARRAY1) {(std::vector<int> * dst),(std::vector<int> * src1),(std::vector<int> * src2)};
// %apply (std::vector<int> & INPLACE_ARRAY1) {(std::vector<int> & dst),(std::vector<int> & src1),(std::vector<int> & src2)};
// %apply (std::vector<int> & INPLACE_ARRAY1, int DIM1) {(std::vector<int> & dst, int a),(std::vector<int> & src1, int b),(std::vector<int> & src2, int c)};

%include "vec_ref.h"

Makefile

all:
    rm -f *.so *.o *_wrap.* *.pyc *.gch vec_ref.py
    swig -c++ -python vec_ref.i
    g++ -O0 -g3 -fpic -c vec_ref_wrap.cxx vec_ref.h vec_ref.cpp -I/home/lmckeereid/tools/anaconda3/pkgs/python-3.7.3-h0371630_0/include/python3.7m/
    g++ -O0 -g3 -shared vec_ref_wrap.o vec_ref.o -o _vec_ref.so

tester.py

import vec_ref as vec
import numpy as np

a = np.array([1,2,3], dtype=np.intc)
b = np.array([4,5,6], dtype=np.intc)
c = np.zeros(len(a), dtype=np.intc)

print('---Before---\na:', a)
print('b:', b)
print('c:', c)

vec.add_vec_ref(c,a,b)

print('---After---\na:', a)
print('b:', b)
print('c:', c)

Output:

---Before---
a: [1 2 3]
b: [4 5 6]
c: [0 0 0]
Traceback (most recent call last):
  File "tester.py", line 12, in <module>
    vec.add_vec_ref(c,a,b)
TypeError: in method 'add_vec_ref', argument 1 of type 'std::vector< int,std::allocator< int > > &'

I've tried all the commented out %apply and %template directives found in vec_ref.i, but they didn't work.

Are there some typemaps that I should be including that I'm not?

That's not possible. In C++, you can only create references to objects that actually exist. However, numpy arrays do not contain a `std::vector`. — pschill, Nov 08 '19 at 11:07
Related: https://stackoverflow.com/questions/51466189/swig-c-to-python-vector-problems — Gabriel Devillers, Apr 21 '20 at 16:01

NicholasM · Answer 1 · 2019-11-15T05:19:28.667

4

I agree with @pschill: it’s not possible to get an std::vector without copying data.

One alternative is to use the std::span class template (introduced in C++20), or a similar span class template defined in a library.

Creating a std::span<int> would provide a view of existing data in a numpy array, and provide many convenient member functions (such as operator[], iterators, front(), back(), etc.) in C++.

Creating a span would never copy data from the numpy array.

edited Nov 15 '19 at 05:19

answered Nov 13 '19 at 21:08

NicholasM

4,557
1
20
47

Thanks for providing what I see to be the best alternative (aside from building my own class). – Otherness Nov 18 '19 at 19:41
If I really wanted to use (and modify) a std::vector in my C++ function without copying, what alternatives would I have ? Raw pointer to std::vector ? shared_ptr to std::vector ? – Gabriel Devillers Apr 21 '20 at 16:27
@GabrielDevillers, if I understand your question, if a vector exists and you want to modify it in your function, I would recommend using a *reference* to the vector: `std::vector& v` – NicholasM Apr 21 '20 at 17:40
@NicholasM I meant in an API that I want to wrap using SWIG. I'm asking because I understand that SWIG cannot wrap non-const reference to vectors. – Gabriel Devillers Apr 22 '20 at 07:17
Oh, sorry. I’d suggest you create a new question that focuses on your specific case. – NicholasM Apr 22 '20 at 14:04

score -1 · Answer 2 · edited Dec 24 '19 at 12:17

-1

You can refer to the Facebook faiss library, which achieves what you want to achieve, in a more elegant way By:

Python-specific: numpy array <-> C++ pointer interface (vector)

You can see the code on its Github page.

edited Dec 24 '19 at 12:17

Alexander van Oostenrijk

4,644
3
23
37

answered Dec 24 '19 at 11:57

王永欣

1
1

How to convert numpy arrays to vector& (reference) with SWIG

2 Answers2

Linked