0

I have a library that I've created with PyO3 on my system using Python 3.5.2. The .so file links to the corresponding libpython3.5m file:

$ ldd my_library.so
    linux-vdso.so.1 =>  (0x00007ffffc823000)
    libpython3.5m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0 (0x00007fcac34b0000)
    ...

But if I try to use my library on another system with a different minor version of Python (eg, 3.6.9 or 3.7.3), that library doesn't exist:

$ ldd my_library.so
    linux-vdso.so.1 (0x00007fffefaae000)
    libpython3.5m.so.1.0 => not found
    ...

And therefore, I can't use my library:

$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_library
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory

Is there a way that I can link my library to libpython more generically to be resilient to these minor version changes or at least allow backward-compatibility (build on newer version but allow older versions of Python to also use my library)? Or anything I can do at runtime to let the library still function?

user655321
  • 1,572
  • 2
  • 16
  • 33
  • Python extension modules shouldn't link to libpython at all. Just avoid linking to libpython. The Python symbols will stay unresolved, and the dynamic loader will look them up in whichever python interpreter loads the module. That way you can even use your module with a statically linked Python! – Daniel May 18 '20 at 21:44
  • Is it even possible to instruct PyO3 to not link to libpython? – user655321 May 18 '20 at 22:41

1 Answers1

2

Python extension modules shouldn't link to libpython at all. The Python symbols will stay unresolved, and the dynamic loader will look them up in whichever python interpreter loads the module. That way you can even use your module with a statically linked Python!

rust-cpython has an extension-module feature for this and PyO3 seems to have copied that. You an enable it in your Cargo.toml:

[dependencies]
pyo3 = { version = ..., features = [ "extension-module" ] }
Daniel
  • 15,944
  • 2
  • 54
  • 60
  • 1
    Unfortunately, while I had specified `"extension-module"` in my Cargo.toml, I had done it incorrectly (perhaps according to a convention in an older version of PyO3), so it didn't actually work. I added `features = ["extension-module"]` to my `[dependencies.pyo3]` section, and that did the trick. – user655321 May 19 '20 at 14:39