How to convert a cython char* filled with 0x00 to non-empty byte array

Question

Here is a cython fonction:

cdef struct my_struct_t:
    unsigned int a
    short b
    unsigned char[6] c

cdef void f_my_struct_t(int buff_size, const unsigned char* buf, list output):
    cdef:
        my_struct_t *arr = <my_struct_t *> buf
        list data = []

    for i in range(buff_size / sizeof(my_struct_t)):
        data.append(arr[i])

    output.append({
        "type": "my_struct_t",
        "data": data
    })

This function takes in parameter a buffer that contains a struct my_struct_t and format this struct into json format.

In C, char* is just an array of bytes.
In cython, it is considered as an array containing only ASCII characters.
So, if the first element is a 0x00 alias '\0', it converts the byte array into b''. While if the char array contains only 0x00, it should convert it as b'000000'.

The current result of this function is:

 {
    'type': 'my_struct_t',
    'data': [
      {
        'a': 1,
        'b': 2,
        'c': b'' # Byte array should not be null
      }
    ]
  }

How this dict could be parsed by json.dumps(xxx, indent=4) with en empty byte array? Currently if fails because of this empty bytearray (or maybe just because there is a bytearray?). Current error: TypeError: Object of type bytes is not JSON serializable

How to force cython to properly convert the char* into byte array?

score 0 · Answer 1 · answered Feb 21 '23 at 21:07

It isn't failing because it's empty - it's failing because it's bytes rather than a string. You need to convert it to a string by doing bytes_value.decode("ascii") (or whatever other encoding you may be using.
By default Cython treats char arrays as C strings and stops at the \0 terminator. You can avoid this by slicing when you cast: <bytes>arr.c[:6]

However, this you'll have to do this for the char array element manually. This won't work as part of Cython's automatic conversion from a C struct to a Python dict (which you're currently using).
A byte array contain only 0 values will convert into b'\0\0\0\0\0\0' (and not b'000000' like you claim). The ascii character '0' is not encoded with the numeric value 0

How to convert a cython char* filled with 0x00 to non-empty byte array

1 Answers1