6

Suppose I am embedding the CPython interpreter into a larger program, written in C. The C component of the program occasionally needs to call functions written in Python, supplying callback functions to them as arguments.

Using the CPython extending and embedding APIs, how do I construct a Python "callable" object that wraps a C pointer-to-function, so that I can pass that object to Python code and have the Python code successfully call back into the C code?

Note: this is a revised version of a question originally posted by user dhanasubbu, which I answered, but which was then deleted. I think it was actually a good question, so I have converted what I wrote into a self-answer to my own statement of the question. Alternative answers are welcome.

zwol
  • 135,547
  • 38
  • 252
  • 361
  • ... I would appreciate it if the people closevoting this question as "too broad" would explain why they think so. It seems like a sensibly scoped question to me! – zwol Aug 29 '18 at 19:26
  • I agree. It's got a clear scope and topic and, since you've answered it, we know it's not too broad to answer. If you really think the question is too broad to help people find the answer, it'd be better to edit it to be more specific rather than throwing it out altogether. – divibisan Aug 29 '18 at 19:56

2 Answers2

4

To define an extension type that is “callable” in the sense Python uses that term, you fill the tp_call slot of the type object, which is the C equivalent of the __call__ special method. The function that goes in that slot will be a glue routine that calls the actual C callback. Here’s code for the simplest case, when the C callback takes no arguments and returns nothing.

typedef struct {
    PyObject_HEAD
    /* Type-specific fields go here. */
    void (*cfun)(void);  /* or whatever parameters it actually takes */
} CallbackObj;

static PyObject *Callback_call(PyObject *self, PyObject *args, PyObject *kw)
{
    /* check that no arguments were passed */
    const char no_kwargs[] = { 0 };
    if (!PyArg_ParseTupleAndKeywords(args, kw, "", no_kwargs))
        return 0;

    CallbackObj *cself = (CallbackObj *)self;
    cself->cfun();
    Py_RETURN_NONE;
}

static PyTypeObject CallbackType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "mymodule.Callback",
    .tp_doc = "Callback function passed to foo, bar, and baz.",
    .tp_basicsize = sizeof(CallbackObj),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_new = PyType_GenericNew,
    .tp_call = Callback_call,
};

Instantiate the type object with PyType_Ready as usual. Don’t put it in any module visible to Python, though, because Python code can’t correctly create instances of this type. (Because of this, I haven’t bothered with a tp_init function; just make sure you always initialize ->cfun after creating instances from C, or Callback_call will crash.)

Now, suppose the actual function you need to call is named real_callback, and the Python function that you want to pass it to is named function_to_call. First you create one of the callback objects, by calling the type object, as usual, and initialize its ->cfun field:

    PyObject *args = PyTuple_New(0);
    CallbackObj *cb = (CallbackObj *)PyObject_CallObject(
        (PyObject *)CallbackType, args);
    Py_DECREF(args);
    cb->cfun = real_callback;

Then you put cb into an argument tuple, and call the Python function object with that, as usual.

    args = Py_BuildValue("(O)", cb);
    PyObject *ret = PyObject_CallObject(function_to_call, args);
    Py_DECREF(args);
    Py_DECREF(cb);
    // do stuff with ret, here, perhaps
    Py_DECREF(ret);

Extending this to more complex cases, where the C callback needs to take arguments and/or return values and/or raise Python exceptions on error and/or receive “closure” information from the outer context, is left as an exercise.

zwol
  • 135,547
  • 38
  • 252
  • 361
2

I'd be tempted to use the standard library ctypes module since it already contains appropriate wrappers for C function pointers, and can automatically deal with conversions from Python types to C types for wide variety of arguments.

I've written a working example in Cython since it's an easy way of mixing Python and C, but it should show how to use these objects:

cdef extern from *:
    """
    int f(int x) {
       return x*2;
    }
    """
    int f(int f)

I define an example function f (in the docstring, which Cython incorporates directly into the compiled file).

import ctypes
from libc.stdint cimport intptr_t

def make_f_wrapper():
    func_type = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int)
    cdef intptr_t f_ptr = <intptr_t>&f
    return func_type(f_ptr)

This is a Cython (pretty close to Python syntax) version of the creation of a ctypes pointer to f. I define the arguments of the function, convert the f pointer to an appropriately sized integer, then initialize the wrapper object with that integer.

cdef extern from *:
    """
    PyObject* make_f_wrapper_c_impl() {
        PyObject *ctypes_module = NULL, *CFUNCTYPE = NULL, *c_int = NULL, 
                 *func_type = NULL, *ptr_value = NULL, *result = NULL;

        ctypes_module = PyImport_ImportModule("ctypes");
        if (ctypes_module == NULL) goto cleanup;
        CFUNCTYPE = PyObject_GetAttrString(ctypes_module,"CFUNCTYPE");
        if (CFUNCTYPE == NULL) goto cleanup;
        c_int = PyObject_GetAttrString(ctypes_module,"c_int");
        if (c_int == NULL) goto cleanup;
        func_type = PyObject_CallFunctionObjArgs(CFUNCTYPE,c_int,c_int,NULL);
        if (func_type == NULL) goto cleanup;
        ptr_value = PyLong_FromVoidPtr(&f);
        if (ptr_value == NULL) goto cleanup;
        result = PyObject_CallFunctionObjArgs(func_type,ptr_value,NULL);

        cleanup:
        Py_XDECREF(ptr_value);
        Py_XDECREF(func_type);
        Py_XDECREF(c_int);
        Py_XDECREF(CFUNCTYPE);
        Py_XDECREF(ctypes_module);
        return result;
    }
    """
    object make_f_wrapper_c_impl()

def make_f_wrapper_c():
    return make_f_wrapper_c_impl()

The code above is a C translation of the Pythony code above - it does exactly the same thing but is a bit more convoluted since it uses the C API. It just uses the ctypes module through its Python interface. Once again the C code is embedded in a Cython file through a docstring; however similar code could be used in a directly written C API module.

(All these Cython snippets combine to form one long working example)

DavidW
  • 29,336
  • 6
  • 55
  • 86