8

Initial situation:

I have a C/C++ library and provide bindings for different target languages. One of these languages is Python. I decided to use Swig because it provides good support for a lot of languages and its integration into CMake is straightforward.

In order to improve the usability of the Python binding and it's installation I would like to provide a setup.py. The idea is to run CMake from within the setup.py file because CMake contains all the logic of how to create the binding correctly.

What I currently got:

I'm able to run CMake and build the target of the Python binding from within setup.py.

By design of Swig the CMake file needs to distinguish between Python 2 and Python 3. While running CMake it detects the location of the Python installation and configures the environment. If the user installed both, Python 2 and Python 3 (with their development packages) CMake always takes Python 3.

According to: https://cmake.org/cmake/help/v3.0/module/FindPythonLibs.html the actual used Python version can be specified by setting PYTHON_LIBRARY and PYTHON_INCLUDE_DIR. For example I have to run:

cmake -DPYTHON_INCLUDE_DIR=/usr/include/python2.7 -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython2.7.so

What is the problem:

If the user executes:

python setup.py build

the Python version of the executable 'python' might be 2 but CMake builds the binding using version 3 (see above).

According to Find python header path from within python? I'm able to get the location of the header file by using:

from distutils.sysconfig import get_python_inc
get_python_inc()

Unfortunately, I don't know how to get the path of the Python library. The solution provided by: Distribution independent libpython path doesn't work for me because it always returns '/usr/lib' instead of '/usr/lib/x86_64-linux-gnu/libpython2.7.so'

The Question:

How do I get the location (the full path) of the Python library from within Python.

Community
  • 1
  • 1
Marcel
  • 203
  • 2
  • 8

3 Answers3

4

The Question:

How do I get the location (the full path) of the Python library from within Python.

The Answer:

Looking for INSTALL_SHARED in the Makefile for python in: https://github.com/python/cpython/blob/3.5/Makefile.pre.in

We can find:

$(INSTALL_SHARED) libpython$(VERSION)$(SO) $(DESTDIR)$(LIBDIR)/$(INSTSONAME); \

We can see there is no single variable with the asked information. You need to build it from two others:

from distutils.sysconfig import get_config_var
libpython = "%s/%s" % (get_config_var("LIBDIR"), get_config_var("INSTSONAME"))

Testing in python 2.7:

Python 2.7.13 (default, Nov 20 2017, 02:33:01)
[GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.sysconfig import get_config_var
>>> "%s/%s" % (get_config_var("LIBDIR"), get_config_var("INSTSONAME"))
'/usr/lib64/libpython2.7.so.1.0'

Testing in python 3.3:

Python 3.3.0 (default, Nov 11 2013, 09:56:47)
[GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.sysconfig import get_config_var
>>> "%s/%s" % (get_config_var("LIBDIR"), get_config_var("INSTSONAME"))
'/usr/lib64/libpython3.3m.so.1.0'

So, it just works for CPython (>=2.3) in linux as you can see in the python Makefile and there is no guarantee that it will work on the future, anyway, this hasn't changed in the last 14 years, since 11th May 2003.

You may use the BINDIR and DLLLIBRARY variables for Windows instead, as you can see from the Makefile:

$(INSTALL_SHARED) libpython$(VERSION)$(SO) $(DESTDIR)$(BINDIR); \

However, I have no means to test in Windows.

olivecoder
  • 2,858
  • 23
  • 22
  • 1
    sysconfig.get_python_lib() returns '/usr/lib/python2.7/dist-packages' which does not contain the file 'libpython2.7.so' – Marcel Nov 04 '15 at 09:38
  • Sorry @Marcel, I didn't pay enough attention to the question. It should be ok now. – olivecoder Nov 04 '15 at 09:49
  • 2
    sysconfig.get_config_var("LDLIBRARY") only returns the name of the library but not the full path. I'm expecting something like: '/usr/lib/x86_64-linux-gnu/libpython2.7.so' – Marcel Nov 04 '15 at 09:53
  • 2
    sysconfig.get_config_var("BINLIBDEST") returns '/usr/lib/python2.7' not the full path to 'libpython2.7.so'. Moreover, '/usr/lib/python2.7' does not contain the 'libpython2.7.so' file – Marcel Nov 04 '15 at 10:22
  • I don't get why @Marcel complains about LDLIBRARY and BINLIBDEST as I have not mentioned those. Furthermore, the above solution still works and it is the right answer for his question: "How do I get the location (the full path) of the Python library from within Python". Please state your reason to downvote then I can have an opportunity to learn or improve my answer. – olivecoder Nov 23 '17 at 14:01
4

I don't know Python a lot and I didn't manage to find a good answer to that problem of figuring out the full path and name of the Python's library. So I tried to come up with a solution by making tests using:

  • CentOS 7 (Python 3.6)
  • Ubuntu 19.10 (Python 3.7)
  • Windows 10 (Python 3.8 from Chocolatey)

I used CMake's find_package(Python3) as a reference, and these are the libraries it found on each of those platforms:

  • CentOS: /usr/lib64/libpython3.6m.so
  • Ubuntu: /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.so
  • Windows: C:/Python38/libs/python38.lib

From that, I tried to find out if there was any combination of configuration variables (as reported by distutils.sysconfig.get_config_vars()) that would give me those paths and library names CMake found. It seemed to me that unfortunately there is no single way that works portably across all these platforms to get the full library path. What I could get of it is that these combinations seem to work on each platform:

  • CentOS: LIBDIR + LDLIBRARY
  • Ubuntu: LIBPL + LDLIBRARY
  • Windows: BINDIR + hardcoding libs\python3.lib
TManhente
  • 2,526
  • 1
  • 15
  • 11
0

I solved the problem on my own with a workaround. Maybe it helps others facing the same or at least a similar problem.

First of all, I introduced the environment variables PYTHON_INCLUDE_DIR and PYTHON_LIBRARY which have the same names as the parameters from CMake used to configure the python paths: -DPYTHON_INCLUDE_DIR and -DPYTHON_LIBRARY. Furthermore, I added the following code to my CMake file

if(DEFINED ENV{PYTHON_LIBRARY} AND DEFINED ENV{PYTHON_INCLUDE_DIR})
    message(STATUS "Using environment variables PYTHON_LIBRARY and PYTHON_INCLUDE_DIR")
    set(PYTHON_LIBRARY $ENV{PYTHON_LIBRARY})
    set(PYTHON_INCLUDE_DIR $ENV{PYTHON_INCLUDE_DIR})
endif()

which simply wraps the environment variables to the CMake parameters. Now the user can define these variables instead of the parameters before running CMake.

PYTHON_INCLUDE_DIR=/usr/include/python2.7 PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython2.7.so cmake

Due to the reason that my setup.py simply calls CMake and CMake now is able to grab the environment variables and wrap them to the parameters respectively, the user also can configure the variables before running the setup.py (or pip, etc.).

PYTHON_INCLUDE_DIR=/usr/include/python2.7 PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython2.7.so pip install tinyspline

To be honest, the user still needs to configure some paths but finally he/she is able to do so.

Marcel
  • 203
  • 2
  • 8
  • 1
    This doesn't really address the problem of finding out the path. You still need to specify it by hand, just in another place. – ivan_pozdeev Nov 22 '17 at 03:13
  • Thist doensn't answer your question: "How do I get the location (the full path) of the Python library from within Python" – olivecoder Nov 23 '17 at 14:02