Python C Extension Exposing a Capsule to ctypes in order to use third party C code

Question

I have a Python C Extension that wraps the library for a proprietary product. Our company has a large amount of C code that uses the proprietary product. Instead of rewriting this in Python using my C Extension, I figured I could simply return a Capsule to Python land, and allow the user of my library to wrap some C function with ctypes.

Is this a valid approach? Is there a better one?

Here is some code to illustrate my approach.

My Python C Extension:

typedef struct {
    PyObject_HEAD

    Foo *foo; /* The proprietary data structure we are wrapping */
} PyFoo;

/*
* Expose a pointer to Foo such that ctypes can use it
*/
static PyObject PyFoo_capsule(PyFoo *self, PyObject *args, PyObject *kwargs)
{
     return PyCapsule_New(self->foo, "foo", NULL);
}

Here is some pre-existing C code our team has written, and wants to call from Python:

void print_foo(Foo *foo)
{
    Foo_print(foo);
}

And in Python, we can wrap the third party C code with ctypes (I learned this here):

import pyfoo
import ctypes

foo = pyfoo.Foo()
capsule = foo.capsule()

ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]
pointer = ctypes.pythonapi.PyCapsule_GetPointer(
    capsule, 
    ctypes.create_string_buffer("foo".encode())
)

libfoo = ctypes.CDLL('libfoo.so')
libfoo.print_foo.restype = None
libfoo.print_foo.argtypes = [ctypes.POINTER(None)]
libfoo.print_foo(pointer)

Wim Lavrijsen · Accepted Answer · 2020-01-24T17:08:13.583

It will work, but what I dislike about the approach of using void*s for opaque types, is that any void* will do, whereas on the C side, types are important and your diagnosis is most likely a segfault (or worse) if a pointer to the wrong type is passed.

Most (automatic) binders (SWIG, pybind11, cppyy for C/C++, or CFFI for C) will generate Python types for the opaque C/C++ ones, to allow type matching.

Here's a cppyy (http://cppyy.org) example, assuming file foo.h like this:

struct Foo;
struct Bar;

typedef Foo* FOOHANDLE;
typedef Bar* BARHANDLE;

void use_foo(FOOHANDLE);
void use_bar(BARHANDLE);

and some matching library libfoo.so, then when used from cppyy, you can only pass FOOHANDLE through FOOHANDLE arguments etc., so that you get a clean Python-side traceback, instead of a C-side crash. Example session:

>>> import cppyy
>>> cppyy.c_include("foo.h")     # assumes C, otherwise use 'include'
>>> cppyy.load_library("libfoo")
>>> foo = cppyy.gbl.FOOHANDLE()  # nullptr; can also take an address
>>> cppyy.gbl.use_foo(foo)       # works fine
>>> cppyy.gbl.use_bar(foo)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: void ::use_bar(Bar*) =>
    TypeError: could not convert argument 1
>>>

EDIT: With a bit of work, the same can be done with ctypes as show below, and thus if you expose a C function that returns self->foo as a Foo* you can likewise annotate its restype with Python FOOHANDLE, thus bypassing capsules and remaining type safe:

import ctypes

libfoo = ctypes.CDLL('./libfoo.so')

class Foo(ctypes.Structure):
    _fields_ = []

FOOHANDLE = ctypes.POINTER(Foo)

class Bar(ctypes.Structure):
    _fields_ = []

BARHANDLE = ctypes.POINTER(Bar)

libfoo.use_foo.restype = None
libfoo.use_foo.argtypes = [FOOHANDLE]

libfoo.use_bar.restype = None
libfoo.use_bar.argtypes = [BARHANDLE]

foo = FOOHANDLE()

libfoo.use_foo(foo)  # succeeds
libfoo.use_bar(foo)  # proper python TypeError

Ignoring performance, can you think of an example when it would be better to write a C Extension than using cffi/cppyy? I've spent the weekend diving into ctypes and cffi, and now I'm wondering if I should have used this instead of writing a massive C extension. Granted I have learned a lot about CPython internals but now I'm thinking I should trash it and start over with cffi. — Matthew Moisen, Jan 27 '20 at 02:09
Using the C-API directly saves a dependency. (But all extension code needs to be recompiled for each version of Python, so a full software stack of extension modules is actually harder to maintain. Note that cffi is very light-weight; but cppyy pulls in LLVM as a dependency.) Also, you may hit a corner case that the binders don't handle, then the C-API could be used for a workaround. Performance-wise, ctypes is rather slow and using the C-API on PyPy can incur a huge penalty (cffi and cppyy OTOH are JIT-friendly). — Wim Lavrijsen, Jan 27 '20 at 04:54
Your edit says "if you expose a C function that returns self->foo as a Foo* you can likewise annotate its restype with Python FOOHANDLE, thus bypassing capsules and remaining type safe" -- Sorry I am missing it- How can my Python C Extension expose the naked Foo* pointer without the capsule? In your example you are instantiating a Foo* pointer using `ctypes.POINTER(Foo)()` - but I need to pass it from the Python C Extension into Python, and then pass it from Python into another C library via ctypes. — Matthew Moisen, Feb 17 '20 at 02:32

Python C Extension Exposing a Capsule to ctypes in order to use third party C code

1 Answers1