Sending big amount of bytes from Swift to Python with types and @_cdecl

Question

I have two simple functions, one pass to another array of UInts. When I pass small array of 20 UInts function works, but when I pass 21576 Uints function returns small amount of bites, why is it happened?

I checked UnsafeMutablePointer<UInt8> inside have correct numbers, but on Python side they are lost.

Swift:

@_cdecl("getPointer")
public func getPointer() -> UnsafeMutablePointer<UInt8>{
    let arr: Array<UInt8> =[1,2,3.....] //here is big array
    if let buffer = buffer {
        buffer.deallocate()
        buffer.deinitialize(count: arr.count)
    }
    buffer = UnsafeMutablePointer<UInt8>.allocate(capacity: arr.count * MemoryLayout<UInt8>.stride)
    buffer!.initialize(from: arr, count: arr.count) 
 
    return buffer!
}

Python:

native_lib = ctypes.CDLL('./libH264_decoder')
native_lib.getPointer.restype = ndpointer(dtype=ctypes.c_uint8)

cont = cast(native_lib.getPointer(), c_char_p).value

returns b'\x1cri\x1aVL\xa4q\xfc\xa7\xaezb\x83HC\x94\xb4#\xde?x\xdb\xb1\xd3\x1d\x07\xb5@\xc8\x85\x0eP\xaa\x9ew\x03\x93\xfe8\xa6\x97D\xca\xc6\xcc'

score 1 · Answer 1 · answered Sep 21 '22 at 10:40

1

On the python caller site, you cast the returned array to a "pointer of chars", which expects to be NULL-terminated. If you do not have any binary zeros in your array, you could check the follwing (only the important parts here):

    var arr: Array<UInt8> = [] //here is big array
    
    for _ in 0..<100 {
        for i in 0..<26 {
            arr.append(UInt8(65+i)) // A...Z
        }
    }
    arr.append(0) // Terminate C-String alike with binary Zero

    buffer = UnsafeMutablePointer<UInt8>.allocate(capacity: arr.count * MemoryLayout<UInt8>.stride)
    buffer!.initialize(from: arr, count: arr.count) 
 
    return buffer!

answered Sep 21 '22 at 10:40

Andreas Oetjen

9,889
1
24
34

Your code works on your array, but on mine from 21577 ints I got only 29 – PrettyLittleHorse Sep 21 '22 at 11:15
If there is a "\x00" (binary zero) in the array, the python-cast stops there. In that case, you need to transfer your array differently (not as c_char_p). Maybe "pythonkit" in swift might help you. – Andreas Oetjen Sep 21 '22 at 11:18
Thank you very much! Is it strange that int 200 converts to \x00? With Python bytes() too – PrettyLittleHorse Sep 21 '22 at 11:40
No, it's not strange. Int(200) consists of four bytes: 0x000000c8 -> contains 3 binary "0" chars. – Andreas Oetjen Sep 21 '22 at 11:46
when I try to cast to another type cast(native_lib.getPointer(), ndpointer(c_int8)).value it returns just 1 integer (140402659470848) and its different every iteration – PrettyLittleHorse Sep 21 '22 at 15:30
maybe I need to use another operator, not "cast" – PrettyLittleHorse Sep 21 '22 at 16:02
I guess it's not that easy. In addition to the data, you somehow need to return the size of the array, otherwise the caller will not be able to determine how much data to expect. I guess there are swift/python interfacing APIs available somehow, but am not sure. – Andreas Oetjen Sep 21 '22 at 19:17

Mark Tolonen · Accepted Answer · 2022-09-22T04:17:45.660

I don't know Swift, but to return a data buffer containing nulls to ctypes you need to know the size of the buffer and can't use c_char_p as the return type since ctypes assumes null-terminated data and converts that specific type to a bytes object. Use POINTER(c_char) instead for arbitrary data that can contain nulls.

Below I've made a simple C DLL that returns a pointer to some data and returns the size in an additional output parameter. The same technique should work for Swift assuming it uses the standard C ABI to export functions, but you will need to pass back both a pointer and a size if the size is variable.

test.c

__declspec(dllexport)
char* get_data(int* size) {
    *size = 8;
    return "\x11\x22\x00\x33\x44\x00\x55\x66";
}

test.py

import ctypes as ct

dll = ct.CDLL('./test')
dll.get_data.argtypes = ct.POINTER(ct.c_int),  
dll.get_data.restype = ct.POINTER(ct.c_char)   # do NOT use ct.c_char_p

size = ct.c_int()  # allocate ctypes storage for the output parameter.
buf = dll.get_data(ct.byref(size))  # pass by reference.
print(buf[:size.value].hex(' '))    # Use string slicing to control the size.
                                    # .hex(' ') for pretty-printing the data.

Output:

11 22 00 33 44 00 55 66

Sending big amount of bytes from Swift to Python with types and @_cdecl

2 Answers2