4

I have a Python package containing a number of C/C++ extensions built as a single wheel. I'm trying to understand how to ensure the wheel and shared libraries it contains correctly advertise that they use the stable ABI at a particular API version. I build the package using a setup.py that I run this way.

% python setup.py bdist_wheel --py-limited-api=cp34

I think the cp34 part is how I indicate that I'm using the stable ABI and at most the Python 3.4 API. The resulting wheel is named goober-1.2-cp34-abi3-linux_x86_64.whl. The highlighted part shows the Python and ABI tags. Without the --py-limited-api, that part is cp38-cp38, matching my Python 3.8. Is that enough to advertise that my wheel should work with all Python 3.x starting from 3.4, without recompiling? I guess I'd specify cp3 to indicate all 3.x versions.

For the shared libraries, I compile the C/C++ source this way.

% gcc ... -DPy_LIMITED_API=0x03040000 ... blooper.c

In this case, the shared library is named blooper.cpython-38-x86_64-linux-gnu.so, with nothing indicating it supports the stable ABI and the 3.4 API. From PEP 3149 I expected to see that somewhere in the name. Otherwise, won't Python 3.8 be the only version willing to import this module?

Thanks.

Jim
  • 474
  • 5
  • 17
  • I guess if you compile the extensions without the aid of setuptools/distutils, there is no way the distribution process can infer the ABI version. So there must be a way (which you already found with the --py-limited-abi flag) to specify the ABI version directly while invoking setptools,distutils,wheel chain. – marscher Sep 04 '21 at 06:50
  • I'm not sure I follow. I'm using setuptools/distutils. I just don't see any hint in the shared libraries that they're supposed to work with Python 3.4, 3.5,... I do see those hints on the wheel file name, and am guessing that's enough for packagers, etc. to know the wheel's applicability. Without something similar on the shared library file names, how can a particular Python version know if it can/should import them? Thanks for the reply. – Jim Sep 04 '21 at 15:25

1 Answers1

3

It might be surprising, but adding --py-limited-api=cp34 only changes the name of the wheel, but not its content - i.e. it will be still "usual" version pinned to the Python version which with it has been built.

The first step is to create a setup.py which would produce a C-extension which uses stable-API and which declares it as well. To my knowledge distutils has no support for stable C-API, so setuptools should be used.

There is a minimal example:

from setuptools import setup, Extension

my_extension = Extension(
            name='foo',
            sources = ["foo.c"],
            py_limited_api = True,
            define_macros=[('Py_LIMITED_API', '0x03040000')],
)

kwargs = {
      'name':'foo',
      'version':'0.1.0',
      'ext_modules':  [my_extension],
}

setup(**kwargs)

Important details are:

  • py_limited_api should be set to True, thus the resulting extension will have the correct suffixes (e.g. abi3), once build.
  • Py_LIMITED_API macro should be set to the correct value, otherwise non-stable or wrong stable C-API will be used.

The resulting suffix of the extension might also be surprising. The CPython documentation states:

On some platforms, Python will look for and load shared library files named with the abi3 tag (e.g. mymodule.abi3.so). It does not check if such extensions conform to a Stable ABI. The user (or their packaging tools) need to ensure that, for example, extensions built with the 3.10+ Limited API are not installed for lower versions of Python.

"Some platforms" are Linux and MacOS, one can check it by looking at

from importlib.machinery import EXTENSION_SUFFIXES
print(EXTENSION_SUFFIXES)
# ['.cpython-38m-x86_64-linux-gnu.so', '.abi3.so', '.so'] on Linux
# ['.cp38-win_amd64.pyd', '.pyd']                         on Windows

that means on Linux, the result will be foo.abi3.so and just foo.pyx on Windows (see e.g. this code in setuptools).

Now, just running

python setup.py bdist_wheel

would build an extension, which could be used with any Python version>= 3.4, but pip would not install it for anything else than CPython-3.8 with pymalloc on Linux (because the name of wheel is foo-0.1.0-cp38-cp38m-linux_x86_64.whl). This is the part, from the documentation, where the packaging system needs to ensure, that it doesn't come to version mismatch.

To allow pip to install for multiple python versions, the wheel should be created with --py-limited-api-version:

python setup.py bdist_wheel --py-limited-api=cp34

due to the resulting name foo-0.1.0-cp34-abi3-linux_x86_64.whl, pip will know, it is safe to install for CPython>=3.4.


To make clear: CPython doesn't really know, that the c-extension with suffix abi3.so (or .pyx on Windows) can really be used by the interpreter (it just assumes in good faith) - it is pip who ensures, that right version is installed.

ead
  • 32,758
  • 6
  • 90
  • 153
  • That answers it for me. The missing piece was the `py_limited_api` key in *setup.py*. With that, my shared library changes from *blooper.cpython-38-x86_64-linux-gnu.so* to *blooper.abi3.so*. That drops more than I expected, but finally makes sense. Thank you. – Jim Sep 06 '21 at 15:15
  • The explanation of `importlib.machinery.EXTENSION_SUFFIXES` was also helpful. – Jim Sep 06 '21 at 15:22