python pass to C-API structure with a pointer to another structure as an element

Question

What is the best way to pass from python 3.6.9 to C-API struct
I have a C library, which I'm trying to create a Python interface for it,
but the library expects to initialize a struct beforehand, the main problem is that one element of a struct is a pointer to another struct.

The part of the C header with the structs:

enum filter_type {
    BLACK,      /* Black list */
    WHITE       /* White list */
};

struct message {
        uint32_t    mid;       /* the message id */
        uint8_t     interface; /* the interface */
} __attribute__ ((__packed__));

struct filtering {
    struct message      *filter;
    uint32_t            arr_len;    /* length of the array
    enum filter_type    mf_type;    /* filter type - black or white list */
} __attribute__ ((__packed__));

Python code:

from ctypes import Structure, c_uint32, c_uint8, c_bool
from myC_Module import test_struct


class Message(Structure):
      _fields_ = [('mid', c_uint32),
                  ('interface', c_uint8)]


def gen_filter(mids, mf_type):
      class Filtering(Structure):
            _fields_ = [('filter', Message*len(mids)),
                        ('arr_len', c_uint32),
                        ('mf_type', c_bool)]

      c_array = Message * len(mids)
      return Filtering(c_array(*mids), len(mids), mf_type)


messages = [  # for testing
      Message(int("1af", 16), 3),
      Message(int("aaaaaaaa", 16), 100),
      Message(int("bbbbbbbb", 16), 200),
]

print(test_struct(gen_filter(messages, True)))

C-API test_struct function code:

static PyObject *test_struct(PyObject *self, PyObject *args) {
    struct filtering *filtering = NULL;
    Py_buffer buffer;
    PyObject *result;

    if (!PyArg_ParseTuple(args, "w*:getargs_w_star", &buffer))
        return NULL;

    printf("buffer.len: %ld\n", buffer.len);
    filtering = buffer.buf;

    printf("recived results: arr_len[%d]  mf_type[%d]\n", filtering->arr_len, filtering->mf_type);
    printf("filter: %d\n", filtering->filter);

    result = PyBytes_FromStringAndSize(buffer.buf, buffer.len);
    PyBuffer_Release(&buffer);
    return result;
}

results:

buffer.len: 32
recived results: arr_len[431]  mf_type[3]
filter: -1442840576
b'\xaf\x01\x00\x00\x03\x00\x00\x00\xaa\xaa\xaa\xaad\x00\x00\x00\xbb\xbb\xbb\xbb\xc8\x00\x00\x00\x03\x00\x00\x00\x01\x00\x00\x00'

With the following approach, in case of a single value of message (not array and by changing the filtering struct accordingly), it works.
Any idea what I'm doing wrong, or What is the right way to this?
Thanks.

EDIT -- Extra Thoughts
I think now I understand why it's happening, yet I don't know how to solve it (yet).
I added additional prints for the buffer.buf to see what I actually have in it:

    filtering = buffer.buf;
    char * buf = (char*)buffer.buf;
    for(int i=0; i<buffer.len; i++){
        if(i==0)
            printf("[%d, ", (uint8_t)buf[i]);
        if(i<buffer.len-1)
            printf("%d, ", (uint8_t)buf[i]);
        else
            printf("%d]\n", (uint8_t)buf[i]);
    }

And I got the following:

[175, 1, 0, 0, 3, 0, 0, 0, 170, 170, 170, 170, 100, 0, 0, 0, 187, 187, 187, 187, 200, 0, 0, 0, 3, 0, 0, 0, 1, 0, 0, 0]

The same results are in python print("[{}]".format(', '.join(map(str, returned))))
For this to work I would expect shorter buffer due to the assigned types (c_uint32, c_uint8)
since I have one gen_filter which have 3 Messages, each Messages should be size 5 and the extra data in gen_filter should be also 5, I would expect total size of 20, but as we can see it's much begger.
I notced that the type c_uint8 is acttualy size 4 insted of 1.
for this to work I expect the following result:

[175, 1, 0, 0, 3, 170, 170, 170, 170, 100, 187, 187, 187, 187, 200, 3, 0, 0, 0, 1]

due to:

{[<c_uint32,c_uint8>,<c_uint32,c_uint8>,<c_uint32,c_uint8>],<c_uint32,c_uint8>}

the buffer have element called format which contain the folloing:
format T{(3)T{<I:mid:<B:interface:}:filter:<I:arr_len:<?:mf_type:}

Answer 1

Your problem is (may be?) that the structure returned by gen_filter isn't the same as the one you define in C. The C equivalent of what you define in gen_filter is:

struct filtering {
    struct message      filter[arr_len]; /* NOT ACTUALLY VALID C SINCE arr_len 
                                            ISN'T A CONSTANT */
    uint32_t            arr_len;    /* length of the array
    enum filter_type    mf_type;    /* filter type - black or white list */
} __attribute__ ((__packed__));

This is a single block of memory, containing space for the message within the structure. However, in C the list of messages is allocated separately to the structure and filter only points to it.

Your Python code should probably be something like this:

class Message(Structure):
      _fields_ = [('mid', c_uint32),
                  ('interface', c_uint8)]
      _pack_ = 1 # matches "packed" - an important addition!

class Filtering(Structure):
    _fields_ = [('filter', POINTER(Message)),
                ('arr_len', c_uint32),
                ('mf_type', c_bool)]
    _pack_ = 1 # matches "packed" - an important addition!
    def __init__(self, messages, mf_type):
        self.filter = (Message*len(messages))(*messages)
        self.arr_len = len(messages)
        self.mf_type = mf_type

Be aware that the lifetime of the separately allocated messages array is tied to the lifetime of the Python Filtering instance.

---

It's slightly hard to follow your C code since you use the mysterious and unspecified attribute filtering->int_list. Assuming int_list is actually filter, you're just printing the pointer (interpreted as a signed int), not what it points to.