Extending python with non python aware C library?

Question

I would like to create a module that uses a non python aware C library and ctypes.

According to this documentation about creating a C extension to python, I’m required to provide a module initialization function with the signature PyObject* PyInit_modulename(void). This looks like a significant complication compared to using the wrapper using ctypes that I already have.

So far I tested building an Extension with setpuptools on unix and it worked. It was simply and handy. But it doesn’t work on windows. I use cibuildwheel and github actions to build wheels for all OS including windows. The compilation on Windows fails because the function PyInit_ext is not found.

I thus need to compile my code as a pure DLL (windows shared library). I then can load the library with ctypes. Can this be done with setuptools which is used by cibuildwheel ?

How can I do that ?

Here is my setup.cfg file:

[metadata]
name = hello_chmike
version = 0.0.0
url = https://github.com/chmike/hello
maintainer = Christophe Meessen
maintainer_email = christophe@meessen.net
description = Simple python module with external C library
long_description = file: README.md
long_description_content_type = text/markdown
keywords = Example
license = BSD 3-Clause License
license_file = LICENSE.txt
classifiers =
    License :: OSI Approved :: BSD License
    Programming Language :: Python :: 3
    Operating System :: OS Independent
    Topic :: Utilities

[options]
packages = hello
python_requires = >=3.6

and here is the setup.py file I currently have and which is not good because it uses Extension:

from setuptools import setup, Extension

setup(
  ext_modules=[Extension('hello.ext',
                       ['src/hello.c'],
                       depends=['src/hello.h'],
                       include_dirs=['src'],
              )],
)

The content of the file hello.c is the following:

#ifdef _MSC_VER
#define _CRT_SECURE_NO_WARNINGS 1
#endif
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

// hello return a heap allocated string containing the name appended 
// to "hello " and followed by "!".
char *hello(char *name) {
    char *buf = malloc(7+strlen(name));
    sprintf(buf, "hello %s!", name);
    return buf;
}

If you don't need to do anything in that function, just provide an empty function? I don't really understand this thing but it seems that you didn't provide one. — user202729, Feb 18 '21 at 09:41
@user202729 Indeed. I didn’t provide one. Would it work if I provide a dummy required function ? That seam like a hack but would solve my problem. I’ll try that. — chmike, Feb 18 '21 at 09:45
@user202729 according to [this answer](https://github.com/joerick/cibuildwheel/discussions/602#discussioncomment-380121) on another forum, it’s because on unix and macOS, symbols are implicitly exported. There is thus no compilation error. On Windows, symbols must be explicitly exported and the missing symbol is detected at compilation time. I could define that symbol as a function and it could work. What signature should this function have ? — chmike, Feb 18 '21 at 09:50
I did say that I don't really understand this thing. Read the documentation. — user202729, Feb 18 '21 at 10:02
@user202729 The hack worked. Compilation succeeded. The test failed because the shared library name is not the same as on unix. I guess it ends with .dll. I guess I better create a real python extension for a more portable solution. It is a bit more work than writing a `ctypes` wrapper, but it is cleaner. Thank you for the help. — chmike, Feb 18 '21 at 10:08

score 1 · Answer 1 · answered Feb 19 '21 at 08:37

The only viable solution to get portable code across different OS and using the python building tools and cibuildwheel to get precompiled modules is to create a real python extension.

This means creating a python extension C code that wraps the python agnostic C library. Wrapping means that we create C functions as required by the Python C extension standard that call the python agnostic C functions from my library.

Using ctypes to wrap the python agnostic C library will work when used for a specific OS like linux. But the execution context is so different between OS that trying to make a ctypes wrapper that works on the every OS requires complex hacking and platform switch in the python wrapper.

By using real python extension C code, we avoid this complexity. I wish I knew this when I started.

Extending python with non python aware C library?

1 Answers1