When writing a python extension in C, how does one pass a python function in to a C function?

Question

First off, apologies for the confusing title.

What I am trying to achieve is the following: Suppose I have some function foo which takes a function and an integer as input. e.g.

int foo(int(*func)(), int i) {
    int n = func() + i;
    return n;
}

Now, I'd like to wrap this function in a python extension module. So I start writing my interface:

#include <Python.h>

extern "C" {
    static PyObject* foo(PyObject* self, PyObject* args);
}

static PyMethodDef myMethods[] = {
    {"foo", foo, METH_VARARGS, "Runs foo"},
    {NULL, NULL, 0, NULL}
}

// Define the module
static struct PyModuleDef myModule = {
    PyModuleDef_HEAD_INIT,
    "myModule",
    "A Module",
    -1,
    myMethods
};

// Initialize the module
PyMODINIT_FUNC PyInit_BSPy(void) {
    return PyModule_Create(&myModule);
}

//Include the function
static PyObject* foo(PyObject* self, PyObject* args){
    // Declare variable/function pointer
    int(*bar)(void);
    unsigned int n;

    // Parse the input tuple
    if (!PyArg_ParseTuple(args, ..., &bar, &n)) {
        return NULL;
    }
}

Now, when it comes time to parse the input tuple, I get confused, as I'm not really sure how to parse it. The idea is rather simple: I need to be able to call foo(bar(), n) in python. But I could use some help on how to realize this.

Answer 1

First off, when you have a Python "extension method", implemented in C, and that function receives a Python callable as an argument, here is how you receive the argument, and how you call the callable:

/* this code uses only C features */
static PyObject *
foo(PyObject *self, PyObject *args)
{
    PyObject *cb;    

    // Receive a single argument which can be any Python object
    // note: the object's reference count is NOT increased (but it's pinned by
    // the argument tuple).
    if (!PyArg_ParseTuple(args, "O", &cb)) {
        return 0;
    }
    // determine whether the object is in fact callable
    if (!PyCallable_Check(cb)) {
        PyErr_SetString(PyExc_TypeError, "foo: a callable is required");
        return 0;
    }
    // call it (no arguments supplied)
    // there are a whole bunch of other PyObject_Call* functions for when you want
    // to supply arguments
    PyObject *rv = PyObject_CallObject(cb, 0);
    // if calling it returned 0, must return 0 to propagate the exception
    if (!rv) return 0;
    // otherwise, discard the object returned and return None
    Py_CLEAR(rv);
    Py_RETURN_NONE;
}

The problem with using logic like this to wrap bsp_init is that the pointer to the Python callable is a data pointer. If you passed that pointer directly to bsp_init, bsp_init would attempt to invoke data as machine code and it would crash. If bsp_init passed through a data pointer to the function that it calls, you could work around this with a "glue" procedure:

/* this code also uses only C features */
struct bsp_init_glue_args {
   PyObject *cb;
   PyObject *rv;
};
static void
bsp_init_glue(void *data)
{
   struct bsp_init_glue_args *args = data;
   args->rv = PyObject_CallObject(args->cb, 0);
}

static PyObject *
foo(PyObject *self, PyObject *args)
{
    bsp_init_glue_args ba;
    if (!PyArg_ParseTuple(args, "O", &ba.cb)) {
        return 0;
    }
    if (!PyCallable_Check(ba.cb)) {
        PyErr_SetString(PyExc_TypeError, "foo: a callable is required");
        return 0;
    }
    bsp_init(bsp_init_glue, (void *)&ba, ...);
    if (ba->rv == 0) return 0;
    Py_CLEAR(ba->rv);
    Py_RETURN_NONE;
}

Unfortunately, bsp_init does not have this signature, so you cannot do this. But the alternative interface BSPLib::Classic::Init takes a std::function<void()>, which is an object-oriented wrapper around the pattern above, so you can do this instead:

/* this code requires C++11 */
static PyObject *
foo(PyObject *self, PyObject *args)
{
    PyObject *cb;
    PyObject *rv = 0;
    if (!PyArg_ParseTuple(args, "O", &cb)) {
        return 0;
    }
    if (!PyCallable_Check(cb)) {
        PyErr_SetString(PyExc_TypeError, "foo: a callable is required");
        return 0;
    }

    std::function<void()> closure = [&]() {
       rv = PyObject_CallObject(cb, 0);
    };
    BSPLib::Classic::Init(closure, ...);

    if (rv == 0) return 0;
    Py_CLEAR(rv);
    Py_RETURN_NONE;
}

The magic here is all in the [&]() { ... } notation, which is syntactic sugar for defining and creating an instance of a local class that "captures" the variables cb and rv so that the code inside the curly braces, which will be compiled as a separate function, can communicate with foo proper. This is a C++11 feature called "lambdas", which is a jargon term going all the way back to the earliest days of theoretical CS and immortalized by Lisp. Here is a tutorial, but I am not sure how good it is because I already know the concept inside and out.

It is not possible to do this in plain C, but it isn't possible to call BSPLib::Classic::Init from plain C either (because you can't define a std::function object at all in plain C ... well, not without reverse engineering the C++ standard library and ABI, anyway) so that's okay.

Answer 2

The Python callable passed in will be a Python object which matches the Callable protocol. Therefore, in order to pass it to your C function, you'll have to create another C function as a proxy which matches the required signature of your function pointer.

As an example:

static PyObject* foo_cb_callable;
static int py_foo_callback(void) {
    PyObject* retval;
    int result;

    // Call the python function/object saved below
    retval = PyObject_CallObject(foo_cb_callable, NULL);

    // Convert the returned object to an int if possible
    if (retval && PyInt_Check(retval))
        result = (int)PyInt_AsLong(retval);
    else
        result = -1;
    Py_XDECREF(retval);
    return result;
}

// NOTE: I renamed this to avoid conflicting with your "foo"
// function to be called externally.
static PyObject* py_foo(PyObject* self, PyObject* args) {
    unsigned int n;
    int result;

    // Parse the input tuple
    if (!PyArg_ParseTuple(args, "OI", &foo_cb_callable, &n)) {
        return NULL;
    }
    // Ensure the first parameter is a callable (e.g. function)
    if (!PyCallable_Check(foo_cb_callable)) {
        return NULL;
    }

    // Call foo with our custom wrapper
    result = foo(&py_foo_callback, n);
    return Py_BuildValue("i", result);
}

Note that I used a global callback object pointer in the example, since your original function pointer didn't have a place for custom user data. If you add a generic user parameter to the func callback, the wrap_foo_callback could be passed to it that way instead of making a global variable. For example:

int foo(int(*func)(void*), void* user_data, int i) {
    int n = func(user_data) + i;
    return n;
}

// ...

static int py_foo_callback(void* callback) {
    PyObject* retval;
    int result;

    // Call the python function/object saved below
    retval = PyObject_CallObject((PyObject*)callback, NULL);

    // Convert the returned object to an int if possible
    if (retval && PyInt_Check(retval))
        result = (int)PyInt_AsLong(retval);
    else
        result = -1;
    Py_XDECREF(retval);
    return result;
}

static PyObject* py_foo(PyObject* self, PyObject* args) {
    PyObject* callback;
    unsigned int n;
    int result;

    // Parse the input tuple
    if (!PyArg_ParseTuple(args, "OI", &callback, &n)) {
        return NULL;
    }
    // Ensure the first parameter is a callable (e.g. function)
    if (!PyCallable_Check(callback)) {
        return NULL;
    }

    // Call foo with our custom wrapper
    result = foo(&py_foo_callback, callback, n);
    return Py_BuildValue("i", result);
}