PicklingError: Can't pickle : attribute lookup c_char_Array_X on ctypes failed

Question

There is problem using ctypes structures with multiprocessing

I can use simple ctypes variables with multiprocessing, but when I use structures passing to funcs there is problem with pickling it

Here is some code which demonstrate this problem

import concurrent.futures
from ctypes import *


def test_c_val(c_val):
    print(c_val.value)
    return c_val.value

test_int = c_int(55)
test_char = c_char(str(6).encode())
arr = [str(i).encode() for i in range(4)]
test_c_array = (c_char * len(arr))(*arr)

futures = []
with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
    futures.append(executor.submit(test_c_val, test_int))
    futures.append(executor.submit(test_c_val, test_char))
    futures.append(executor.submit(test_c_val, test_c_array))
    time.sleep(5)
    print(futures[2])
    
print(futures)
print(futures[2].exception())

How can i solve it?

[\[SO\]: Pickling a ctypes.Structure with ct.Pointer (@CristiFati's answer)](https://stackoverflow.com/a/72525657/4788546). — CristiFati, Jan 11 '23 at 16:25

Ahmed AEK · Answer 1 · 2023-01-11T16:54:56.170

ctypes pointers can point to memory of any size or location that python doesn't know about, it may even point to null, and it's unsafe for python to attempt to pickle it, so python doesn't pickle it.

the closest thing is to either have a shared memory array using multiprocessing.Array or using python array which can be pickled.

this is how it would be done using an array, as it is a python object that can be pickled, the array is sent and a pointer to it is used to form a ctypes object.

import concurrent.futures
import ctypes
import array
import time

def test_c_val(c_val):
    print(c_val.value)
    return c_val.value

def test_py_array(py_arr:array.array):
    c_array_size = py_arr.buffer_info()[1]
    c_array = (ctypes.c_char*c_array_size).from_address(py_arr.buffer_info()[0])
    print(c_array.value)
    return py_arr

if __name__ == "__main__":
    test_int = ctypes.c_int(55)
    test_char = ctypes.c_char(str(6).encode())
    arr = [str(i).encode() for i in range(4)]
    test_c_array = (ctypes.c_char * len(arr))(*arr)
    # test_py_c_array = array.array('b', b''.join(arr))
    test_py_c_array = array.array('b', test_c_array.value)
    futures = []
    with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
        futures.append(executor.submit(test_c_val, test_int))
        futures.append(executor.submit(test_c_val, test_char))
        futures.append(executor.submit(test_py_array, test_py_c_array))
        time.sleep(1)
        print(futures[2])

    print(futures)
    print(futures[2].exception())

if you want to use shared memory then you can use multiprocessing.shared_memory which is more flexible than multiprocessing.Array, note that you can't pickle multiprocessing.Array, it can only be inherited, unlike python array.

alternatively there's numpy which allows easier and serializable c arrays which are easier to use than python arrays.

TY, looks like solution, but maybe exception should be 'ValueError: ctypes arrays cannot be pickled' as it is when you'r trying pickle ctypes pointers. As for me 'PicklingError: Can't pickle : attribute lookup c_char_Array_X on ctypes failed' is more complicated and confusable than 'ValueError'. Or am I wrong? — sush1, Jan 12 '23 at 06:19
@sush1 not really, what the error is telling you is that you can pickle ctypes pointers if you define your custom implementation of how to pickle it, but it ends up doing what the above code does but in more steps, and it doesn't generalize to any pointer, and you might end up with segfault. — Ahmed AEK, Jan 12 '23 at 07:45

PicklingError: Can't pickle : attribute lookup c_char_Array_X on ctypes failed

1 Answers1