2

Question:
What is the proper way to structure and build a Python wheel with C extensions for deployment? Ideally, the user shouldn't need elevated privileges or be required to point environment variables to the contents of the package.

What I have tried:
My package directory is structured as follows.

project
 |
 |--- foo/
 |    |--- __init__.py
 |    |--- foo.py
 |    |--- libHelper.so
 |
 |--- foo.c
 |--- setup.py

My setup.py looks like the following.

import os
import sys
import glob
from setuptools import setup, find_packages, find_namespace_packages, Extension

application_module = Extension(
    '_foo',
    include_dirs=[],
    libraries=['Helper'],
    library_dirs=['foo'],
    sources=['foo.c'],
    extra_compile_args='',
    extra_link_args=''
)

setup(
    name=PACKAGE_NAME,
    version='1.0.0',
    author="",
    author_email="",
    description='',
    url='',
    ext_modules=[application_module],
    packages=find_packages(),
    package_data={
        '' : ['*.so']
    },
    python_requires='>=3.6'
)

When I execute python setup.py bdist_wheel, the build directory looks like the following:

build/
 |--- lib.linux-x86_64-3.5/
      |--- _foo.cpython-35m-x86_64-linux-gnu.so
      |--- foo/
            |--- __init__.py
            |--- foo.py
            |--- libHelper.so

When I install this wheel, I start python and attempt to import the package.

$ python
Python 3.5.2 (default, Oct  8 2019, 13:06:37)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/somedude/.local/lib/python3.5/site-packages/foo/__init__.py", line 1, in <module>
    from .foo import *
  File "/home/somedude/.local/lib/python3.5/site-packages/foo/foo.py", line 28, in <module>
    _foo = swig_import_helper()
  File "/home/somedude/.local/lib/python3.5/site-packages/foo/foo.py", line 20, in swig_import_helper
    import _foo
ImportError: libHelper.so: cannot open shared object file: No such file or directory

However, just for testing purposes, if I update LD_LIBRARY_PATH, the error goes away.

export LD_LIBRARY_PATH=/home/somedude/.local/lib/python3.5/site-packages/foo/
rgmann
  • 31
  • 2
  • Disclaimer: I am not an expert, but judging from the [documentation](https://docs.python.org/3/distutils/setupscript.html#library-options), I would try setting `runtime_library_dirs` for your extension. – cel Jun 08 '20 at 19:57
  • @cel Thanks! That is an interesting lead. Unfortunately, the `*.so` is included in the wheel, so I would have to somehow point to the wheel directory, which requires knowledge of the user's python environment that I don't have at build time. – rgmann Jun 08 '20 at 20:58
  • 1
    Hmh, in principle it should be possible to find the .so relative to the extension module, see e.g.: https://unix.stackexchange.com/questions/137492/load-shared-objects-relative-to-executable-path. At this point it may be easier though to compile the code that generates `libHelper.so` directly into your extension module if you can ship this code as well. After all `.so` files are shared libraries and you are not at all intending to share them, are you? ;-) – cel Jun 08 '20 at 21:13

0 Answers0