in method 'my_print_hexbytes', argument 1 of type 'uint32_t *'

Question

I'm writing a Python C extension by SWIG but getting frustrated when passing binary buffer to C api functions. Here's my example:

In utils.c

#include "utils.h"
void my_print_hexbytes(uint32_t *bytes, uint32_t bytes_len)
{
  uint32_t i;
  for(i = 0; i < bytes_len; i++)
    printf("%02x", bytes[i]);
  printf("\n");
}

In utils.h

#include "commons.h"
#ifndef XXXX_UTILS_H
#define XXXX_UTILS_H
void my_print_hexbytes(uint32_t *bytes, uint32_t bytes_len);
#endif /* XXXX_UTILS_H */

In commons.h

#ifndef XXXX_COMMONS_H
#define XXXX_COMMONS_H
....
....
#include <stdint.h>
....
....
#endif

In xxxx_for_py.i

%module xxxx_for_py
%{
#define SWIG_FILE_WITH_INIT
#include "commons.h"
#include "utils.h"
%}
%include "stdint.i"
void my_print_hexbytes(uint32_t *bytes, uint32_t bytes_len);

1st Try

After compiling out _xxxx_for_py.so, I gave a try testing it in IPython:

In [1]: import xxxx_for_py

In [2]: from ctypes import *

In [3]: a_t = c_uint * 2

In [4]: a = a_t(0x1, 0x2)

In [5]: xxxx_for_py.my_print_hexbytes(a, 2)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-3c023fcf8b04> in <module>()
----> 1 xxxx_for_py.my_print_hexbytes(a, 2)

TypeError: in method 'my_print_hexbytes', argument 1 of type 'uint32_t *'

That time I had no idea how to handle this case. I tried changing the a as bytearray but it didn't work.

Not sure if anyone who could help me on this issue. Thanks!

2nd Try

Thanks for @Petesh comment. I've hit a try by casting a to POINTER(c_uint) but still not work :(

In [5]: xxxx_for_py.my_print_hexbytes(cast(a, POINTER(c_uint)), 2)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-2abb9dae484d> in <module>()
----> 1 xxxx_for_py.my_print_hexbytes(cast(a, POINTER(c_uint)), 2)

TypeError: in method 'my_print_hexbytes', argument 1 of type 'uint32_t *'

3rd Try

Referenced by Unbounded Array section in SWIG documentation, I added carray.i to xxxx_for_py.i

In xxxx_for_py.i

....
....stdint.i"
%include "carrays.i"
%array_class(uint32_t, uint32Array)
....

After re-compiling and loading .so, it still got same error

In [1]: import xxxx_for_py

In [2]: a = xxxx_for_py.uint32Array(2)

In [3]: a[0] = 0x0

In [4]: a[1] = 0x1

In [5]: xxxx_for_py.my_print_hexbytes(a, 2)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-3c023fcf8b04> in <module>()
----> 1 xxxx_for_py.my_print_hexbytes(a, 2)

TypeError: in method 'my_print_hexbytes', argument 1 of type 'uint32_t *'

4th Try

After a week later at that time, user Nethanel post an answer. Though it still did not work, it could help me find something interesting.

Answer 1

After brief testing, I found solution by myself. See this article for details http://au9ustine.github.io/wiki/crypto/cuda/swig.html#typeerror-in-method-xxxx-argument-y-of-type-zzzz (Update on 2017-01-13: This page is broken, I'll move that page content here)

A brief anatomy

Keeping trying on testing Python codes on passing binary buffer to a trivial C module (i.e. library compiled by gcc without extra dependencies). And internals might look like:

integer list -> array/struct -> raw string -> C function parameter

From some descriptions from create_string_buffer. By create_string_buffer, Python could provide such a feature dynamically allocating pieces of memory for current variables for use.

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

size_t
print_int_buf(void *src, size_t len)
{
    size_t i = 0;
    uint32_t *int_ptr = (uint32_t *)src;
    for (i = 0; i < len; i++) {
        printf("%p: %x\n", int_ptr + i, int_ptr[i]);
    }
    return i;
}

If compiled as a shared library named my_ext_lib.so, the code could be loaded in Python

import ctypes
from ctypes import *
import array

my_ext = ctypes.CDLL('./my_ext_lib.so')
buf = create_string_buffer(array.array('I', range(0x10)).tostring())
my_ext_lib.print_int_buf(buf, 0x10)

And it did work.

Though ctype and SWIG are quite different approaches for interoperating integration between Python and C, I use ctype code here to elaborate the basic mechanism and its working way of passing parameters to a compiled module (both for Python module and trivial C shared library module).

Interim solution

Since the significant difference was the function interface (or method signature) from my view, why not change it to void * instead of uint32_t *? (Though it's not the best way, it could be still an interim solution for containment)

I replaced uint32_t * with void * for the method my_print_hexbytes, and appended dependency cdata.i (I'm not sure if it works but for safety it has been added). So the changes were listed below and eventually it displayed expected result.

utils.c

#include "utils.h"
...
void 
my_print_hexbytes(void *bytes, size_t bytes_len)
{
  size_t i = 0;
  uint32_t *int_buf_ptr = (uint32_t *)bytes;
  for(i = 0; i < bytes_len; i++)
    printf("%02x", int_buf_ptr[i]);
  printf("\n");
}

header utils.h

#include "commons.h"
#ifndef XXXX_UTILS_H
#define XXXX_UTILS_H
...
void my_print_hexbytes(void *bytes, size_t bytes_len);
#endif /* XXXX_UTILS_H */

xxxx_for_py.i

%module xxxx_for_py
%{
#define SWIG_FILE_WITH_INIT
#include "commons.h"
#include "utils.h"
%}
%include "stdint.i"
%include "carrays.i"
%include "cdata.i"
%array_class(uint32_t, uint32Array)
...
void my_print_hexbytes(void *bytes, size_t bytes_len);

Invoking method in Python, it would be like:

In [1]: import xxxx_for_py

In [2]: a = xxxx_for_py.uint32Array(0x10)

In [3]: for i in range(0x10):
   ...:     a[i] = i
   ...:

In [4]: xxxx_for_py.my_print_hexbytes(a, 0x10)
000102030405060708090a0b0c0d0e0f

Answer 2

I had a very simple case but still got this error. The fix from this question worked: Swig python - c++ how to use type int8_t

I added %include "stdint.i" directly after the module def in the .i file

Answer 3

Found you question, because of same problem.

Unfortunately, it seems advanced types are not supported by SWIG: in http://www.swig.org/Doc3.0/Library.html#Library_carrays, after description of array_class macro, it is written: "When using this macro, type is restricted to a simple type name like int or float."

I also tried unsuccessfully to use "unsigned int".

In the end my workaround was to use C++ vectors instead of uint32_t pointers, here is relevant part of swig file:

%include "stdint.i"
%include "std_vector.i"

namespace std {
   %template(vectoruint32) vector<uint32_t>;
};

part of h file:

#include <vector>

static uint32_t foo(std::vector<uint32_t> v) {return v[1] - v[0];}

python:

import example
l = [1, 2, 3, 4]
example.foo(l)

Hope this helps