Sharing Python multiprocessing shared Value with C extension

Question

I have two processes in python that share a single boolean flag:

from multiprocessing import Process, Value

class MyProcess(Process):
   def __init__(self):
       self.flag = Value('B',false)
       # [...]

   def run(self):
       while self.active:
           # do_something()
           if some_condition:
               self.work_to_be_extended__()


   def work_to_be_extended__(self) -> bool:
       while some_internal_loop_condition:
           if self.flag.value:
               # do something
      return result
  
if __name__ == '__main__':
    my_proc = MyProcess()
    my_proc_flag = my_proc.flag
    my_proc.start()
    # [...] Some work
    if condition:
        my_proc_flag.value = True

I need to put MyProcess.work_to_be_extended in an extension module to be executed in C code. Something like:

bool extended_work(void):
{
    while (some_condition) {
        if (my_proc_flag) {
            do_something()
        }
    return result
}

I've not designed the extension yet, since I'd need to understand first how to share the MyProcess.flag variable. Please, note that I don't need to pass the variable value, I need to pass its reference in order for the extension to see a change in the flag value operated in the main process where the extension does not live`.

Hope I've been quite clear**

Answer 1

Multiprocessing has a sharedctypes submodule for ctypes array and values. You can use it to create a shared ctypes (a int in my example). And then use ctypes.byref to send a pointer to that int. Since the underlying mechanism is SHM (not some hidden piping under the hood), the pointed memory by this reference is really the same in both process. shval.value is *p pointed by the p argument passed, that is byref(shval).

So, no need for the size 1 array of my previous answer, and, more importantly, for the disclaimer accompanying it.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>

void myf(volatile uint32_t *p){
    for(;;){
        printf("<c>%d</c>\n", *p);
        if(*p==100) return;
        (*p)++;
        sleep(1);
    }
}

import multiprocessing as mp
import multiprocessing.sharedctypes as st
import ctypes

mylib=ctypes.CDLL("libtoto.so")
mylib.myf.argtypes=[ctypes.c_void_p]
shval=st.RawValue(st.ctypes.c_uint32,12)

class MyProcess(mp.Process):
    def __init__(self):
        super().__init__()

    def run(self):
        mylib.myf(st.ctypes.byref(shval))

if __name__=="__main__":
    myproc=MyProcess()
    myproc.start()
    while True:
        i=int(input("val>"))
        shval.value=i

So, short answer to your question is: use multiprocessing.sharedctypes and pass byref(sharedval) to your function.

Answer 2

Premise

This answer comes from an adaptation of the good solution given by @chrslg. This extends that usage to apply to other paradigm of Python/C programming, such as C Extension API, Cython and Boost::Python.

Please, read that answer first for a deeper background.

Overview and core summary:

Using a sharedctypes.RawValue as the required variable, it is possible to access the underlying data address by means of the method sharedctypes.ctypes.addressof.

Therefore, one can pass the address of the variable as a long long int (64 bit) and cast it into a pointer to the required data. For example, for a uint8_t variable, one has into the C extension

 int64_t address; // This is initialized in some way, depending on the C interface to python
 
 // Pointer to shared data
 uint8_t* pointer = (uint8_t*)(address);

 printf("Current value of shared data: %u\n", pointer);

Working example for different Python - C/C++ interfaces

Common C shared library

Let's create a base, simple C library that just read for 1 time per second the value of the variable being shared:

// cshare_data/cshare_data.c

#include "cshare_data.h"

#include <time.h>
#include <unistd.h>
#include <stdio.h>


void cshare_data(uint8_t* data, char from_where_called) {
    char *s = NULL;
    if (from_where_called == 0) {
        s = "cTypes CDLL";
    } else if (from_where_called == 1)
    {
        s = "Python C Extension";
    } else if (from_where_called == 2)
    {
        s = "Boost::Python";
    } else if (from_where_called == 3)
    {
        s = "Cython";
    }
    for (int i = 0; i < 10; i++) {
        printf("C code read from %s a value of: %u\n", s, *data);
        sleep(1);
    }
}

The header:

// cshare_data/cshare_data.h

#ifndef CSHARE_DATA_H
#define CSHARE_DATA_H

#include <stdint.h>
#include <stddef.h>

extern void cshare_data(uint8_t*, char);

#endif

Python shared data editing process

For the rest of the examples, I'll refer to the following Python process that is modifying the shared data (unsigned char in the example):

from multiprocessing.sharedctypes import RawValue, Value
import multiprocessing.sharedctypes as st
from multiprocessing import Process


class MyProcess(Process):
    def __init__(self):
        Process.__init__(self)
        self.int_val = RawValue(st.ctypes.c_ubyte, 0)
    def run(self) -> None:
        import time
        for _ in range(10):
            print('Value in Python Process: ', self.int_val.value)
            self.int_val.value += 1
            time.sleep(1)

my_proc = MyProcess()
my_proc.start()

NOTE: This will not be repeated hereinafter.

Python C Extension

A Python C Extension API that makes use of the above pattern follows:

#include <Python.h>

#include <stdio.h>
#include <time.h>

#include "cshare_data.h"

static PyObject *cshare_data_wrapper(PyObject *self, PyObject *args)
{
    PyObject *val = NULL;

    // This will store the address of the uchar variable being passed from Python
    int64_t address = 0;

    // Convert the single element tuple into a 8-byte int (address)
    if(!PyArg_ParseTuple(args, "L", &address)) {
        printf("Error parsing Tuple\n");
        return NULL;
    }

    // Now address is reinterpreted as the shared variable pointer
    uint8_t *pointer = (uint8_t *)(address);
    
    // Call the library function
    cshare_data(pointer, 1);

    return Py_None;
}

static PyMethodDef CShapreDataMethods[] = {
    {"cshare_data", cshare_data_wrapper, METH_VARARGS, "Python interface for sharedata C library function"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef cshareddata_module = {
    PyModuleDef_HEAD_INIT,
    "csharedata_module",
    "Python interface for the fputs C library function",
    -1,
    CShapreDataMethods
};

PyMODINIT_FUNC PyInit_cshare_data_pyext(void) {
    return PyModule_Create(&cshareddata_module);
}

Please, refer to official documentation and this very good tutorial for deeper insight about Python C-API

Boost::Python

Very similar to what done for the Python C-API, the boost wrapper looks like:

extern "C" {
    #include "cshare_data.h"
}
#include <boost/python.hpp>

void cshare_data_boost_wrapper(long long int data_address) {
    uint8_t* data = reinterpret_cast<uint8_t*>(data_address);
    cshare_data(data, 2);
}

BOOST_PYTHON_MODULE(ctrigger) {
    using namespace boost::python;
    def("cshare_data", cshare_data_boost_wrapper);
}

CMake - Library buildings

Moving from a project with the following tree structure:

```
project_root
|   cshare_data.py
|---clibs
|   |   cshare_data_boost.so
|   |   cshare_data_pyext.so
|   |   cshare_data.so
|
|---cshare_data
|   |   cshare_data.c
|   |   cshare_data.h
|   
|   CMakeList.txt
```

The following compilation CMake script was used:

cmake_minimum_required (VERSION 2.6)

project (cshare_data)

set(CMAKE_SHARED_MODULE_PREFIX "")
set(CMAKE_SHARED_LIBRARY_PREFIX "")

# Common C shared library

set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_SOURCE_DIR}/clibs)

include_directories(${CMAKE_SOURCE_DIR}/cshare_data)
link_directories(${CMAKE_SOURCE_DIR}/clibs)
# --- Common C shared library ---
add_library(cshare_data SHARED cshare_data/cshare_data.c)

# Needed for Python C Extension Module and Boost::Python
include_directories("/usr/include/python3.8")

# --- Python C Extension Module library ---
add_library(cshare_data_pyext MODULE cshare_data_pyinterface/cshare_data_pyext.c)
target_link_libraries(cshare_data_pyext python3.8)
target_link_libraries(cshare_data_pyext cshare_data)


# --- Python C Extension Module library ---
include_directories("/home/buzz/boost_1_80_0")
link_directories("/home/buzz/boost_1_80_0/build/lib")
add_library(cshare_data_boost MODULE cshare_data_pyinterface/cshare_data_boost.cpp)
target_link_libraries(cshare_data_boost python3.8)
target_link_libraries(cshare_data_boost boost_python38)
target_link_libraries(cshare_data_boost cshare_data)

Python - Calling C wrappers

Just for the purpose of demonstration, I've written 3 different processes that share the same int_val (handled by the above MyProcess) and call the C function to print the value of this variable. Note that, though the lines of code are the same, the address must be withdrawn at each process call since multiprocessing.sharedctypes wraps IPC synchronizing architecture for int_val under the hood, meaning that each actual int_val lives in the proper process.

my_proc = MyProcess()
my_proc.start()

l = []


class FromWhere(IntEnum):
    ctype = 0
    python_c_extension = 1
    boost_python = 2

def from_ctype_import_dll(int_val: RawValue):
    import ctypes
    reference = st.ctypes.byref(my_proc.int_val)
    mylib=ctypes.CDLL("clibs/cshare_data.so")
    mylib.cshare_data.argtypes=[ctypes.c_void_p, ctypes.c_char]
    mylib.cshare_data(reference, FromWhere.ctype.value)


def from_python_c_extension(int_val: RawValue):
    from clibs import cshare_data_pyext
    address = st.ctypes.addressof(int_val)
    cshare_data_pyext.cshare_data(address)

def from_boost_python(int_val: RawValue):
    from clibs import cshare_data_boost
    address = st.ctypes.addressof(int_val)
    cshare_data_boost.cshare_data(address)

ps: List[Process] = []

ps.append(Process(target=from_ctype_import_dll, args=(my_proc.int_val,)))
ps.append(Process(target=from_python_c_extension, args=(my_proc.int_val,)))
ps.append(Process(target=from_boost_python, args=(my_proc.int_val,)))

for p in ps:
    p.start()

for p in ps:
    p.join()

The result achieved:

Value in Python Process:  0
C code read from cTypes CDLL a value of: 1
C code read from Python C Extension a value of: 1
C code read from Boost::Python a value of: 1
Value in Python Process:  1
C code read from cTypes CDLL a value of: 2
C code read from Boost::Python a value of: 2
C code read from Python C Extension a value of: 2
Value in Python Process:  2
C code read from cTypes CDLL a value of: 3
C code read from Boost::Python a value of: 3
C code read from Python C Extension a value of: 3
C code read from cTypes CDLL a value of: 3
Value in Python Process:  3
C code read from Boost::Python a value of: 4
C code read from Python C Extension a value of: 4
C code read from cTypes CDLL a value of: 4
Value in Python Process:  4
C code read from Boost::Python a value of: 5
C code read from Python C Extension a value of: 5
C code read from cTypes CDLL a value of: 5
Value in Python Process:  5
C code read from Boost::Python a value of: 6
C code read from Python C Extension a value of: 6
C code read from cTypes CDLL a value of: 6
Value in Python Process:  6
C code read from Python C Extension a value of: 7
C code read from Boost::Python a value of: 7
C code read from cTypes CDLL a value of: 7
Value in Python Process:  7
C code read from Python C Extension a value of: 8
C code read from Boost::Python a value of: 8
C code read from cTypes CDLL a value of: 8
Value in Python Process:  8
C code read from Python C Extension a value of: 9
C code read from Boost::Python a value of: 9
C code read from cTypes CDLL a value of: 9
Value in Python Process:  9
C code read from Python C Extension a value of: 10
C code read from Boost::Python a value of: 10