Best way to package a Python library that includes a C shared library?

Question

I have written a library whose main functionality is implemented in C (speed is critical), with a thin Python layer around it to deal with the ctypes nastiness.

I'm coming to package it and I'm wondering how I might best go about this. The code it must interface with is a shared library. I have a Makefile which builds the C code and creates the .so file, but I don't know how I compile this via distutils. Should I just call out to make with subprocess by overriding the install command (if so, is install the place for this, or is build more appropriate?)

Update: I want to note that this is not a Python extension. That is, the C library contains no code to itself interact with the Python runtime. Python is making foreign function calls to a straight C shared library.

Have you got any solution for your problem? I'm stuck with the same question and desperate for an answer ;) — MrLeeh, Jul 15 '17 at 15:00
This is a very long time ago but it looks like I solved this by shelling out to `make`. This is the setup.py from that library: https://gist.github.com/obeattie/3d491ec4c29b1d4b46387b285c91ca89 — obeattie, Jul 17 '17 at 14:52
I've tried doing what you have in this gist, but... how did you get the `.so` you lined with to get installed in any location Python / ld will look for it? — wvxvw, Jan 24 '18 at 16:24

score 1 · Answer 1 · answered Jun 05 '14 at 15:29

1

Given that you followed the instructions on how to create Python extensions in C, you should just enlist the extension modules like in this documentation.
So the setup.py script of your library should look like this:

from distutils.core import setup, Extension
setup(
   name='your_python_library',
   version='1.0',
   ext_modules=[Extension('your_c_extension', ['your_c_extension.c'])],
)

and distutils knows how to compile your extension to C shared library and moreover where to put it.

Of course I have no further information about your library, so you probably want to add more arguments to setup(...) call.

answered Jun 05 '14 at 15:29

ElmoVanKielmo

10,907
2
32
46

@ElmoVanKielvo, do you know how to do it in distutils2 ? – Mansueli Jun 05 '14 at 15:38
1

Seems like... http://pythonhosted.org/Distutils2/distutils/setupscript.html#describing-extension-modules ...exactly the same :) – ElmoVanKielmo Jun 05 '14 at 15:42
11

It is not a Python extension. This is a pure C library (which contains no code itself to interact with the Python runtime), but Python makes calls to it via [`ctypes`](https://docs.python.org/2/library/ctypes.html). It's potentially possible to wrap the C shared library in a Python extension module, but I'd prefer to keep the current `ctypes` approach. – obeattie Jun 05 '14 at 17:21

score 1 · Answer 2 · answered Jun 05 '14 at 17:28

1

I'd consider building the python module as a subproject of a normal shared library build. So, use automake, autoconf or something like that to build the shared library, have a python_bindings directory with a setup.py and your python module.

answered Jun 05 '14 at 17:28

Sam Hartman

6,210
3
23
40

score 1 · Answer 3 · edited May 23 '17 at 12:34

I had a similar need and found this answer helpful: Python setup.py call makefile don't include binaries.

I have an ANSI C library in the src directory of my distribution. In the src directory is a Makefile that builds a file called liblsd.so in my package directory (lsd). I call this in setup.py and then tell setup to include the library file using the package_data argument.

import os.path
import subprocess

from setuptools import setup

with open(os.path.join(os.path.dirname(__file__), 'README.rst')) as f:
    readme = f.read()

subprocess.call(['make', '-C', 'src'])

setup(name='LSD',
  version='0.0.1',
  description='Python bindings for the LSD line segment detector.',
  long_description=readme,
  author='Geoff Hing',
  author_email='geoffhing@gmail.com',
  url='https://github.com/ghing/python-lsd',
  packages=['lsd'],
  package_data={'lsd': ['liblsd.so']},
  include_package_data=True,
  classifiers=[
      'Development Status :: 1 - Planning',
      'Intended Audience :: Developers',
      'License :: OSI Approved :: MIT License',
      'Operating System :: OS Independent',
      'Programming Language :: Python',
      'Programming Language :: C',
      ],
 )

You should at least sub class `distutils.command.build` and override the `run` method, otherwise the make call will happen for every type of call to setup.py where you may not want to recompile the library (e.g. 'check' or 'upload') — Tom Close, Oct 21 '16 at 22:34

Best way to package a Python library that includes a C shared library?

3 Answers3

Linked