15

When using setuptools/distutils to build C libraries in Python

$ python setup.py build

the *.so/*.pyd files are placed in build/lib.win32-2.7 (or equivalent).

I'd like to test these files in my test suite, but I'd rather not hard code the build/lib* path. Does anyone know how to pull this path from distutils so I can sys.path.append(build_path) - or is there an even better way to get hold of these files? (without having installed them first)

danodonovan
  • 19,636
  • 10
  • 70
  • 78
  • 1
    I've wanted this on occasion too, but I've never been motivated enough to go looking for an answer ... – mgilson Jan 14 '13 at 14:23

4 Answers4

16

You must get the platform that you are running on and the version of python you are running on and then assemble the name yourself. This is an internal detail of setuptools (based on the bundled version of distutils), and so is not guaranteed to be stable.

To get the current platform, use sysconfig.get_platform(). To get the python version, use sys.version_info (specifically the first three elements of the returned tuple). On my system (64-bit linux with python 2.7.2) I get:

>>> import sysconfig
>>> import sys
>>> sysconfig.get_platform()
linux-x86_64
>>> sys.version_info[:3]
(2, 7, 2)

Before setuptools 62.1

The format of the lib directory is "lib.platform-versionmajor.versionminor" (i.e. only 2.7, not 2.7.2). You can construct this string using python's string formatting methods:

def distutils_dir_name(dname):
    """Returns the name of a distutils build directory"""
    f = "{dirname}.{platform}-{version[0]}.{version[1]}"
    return f.format(dirname=dname,
                    platform=sysconfig.get_platform(),
                    version=sys.version_info)

After setuptools 62.1

The directory above clashes with alternative python implementations like PyPy, so the internal build directory was made more specific using sys.implementation.cache_tag. The format of the lib directory is "lib.<platform>-<cachetag>". You can construct this string using python's string formatting methods:

def distutils_dir_name(dname):
    """Returns the name of a distutils build directory"""
    f = "{dirname}.{platform}-{cache_tag}"
    return f.format(dirname=dname,
                    platform=sysconfig.get_platform(),
                    cache_tag=sys.implementation.cache_tag)

You can use this to generate the name of any of distutils build directory:

>>> import os
# Before Setuptools 62.1
>>> os.path.join('build', distutils_dir_name('lib'))
build/lib.linux-x86_64-3.9
# After Setuptools 62.1
>>> os.path.join('build', distutils_dir_name('lib'))
build/lib.linux-x86_64-cpython39

Note that Setuptools 62.1 is only available on Python 3.7+. Also, if you set SETUPTOOLS_USE_DISTUTILS=stdlib in your environment, then you will get the old behavior even in Setuptools 62.1.

Henry Schreiner
  • 905
  • 1
  • 7
  • 18
SethMMorton
  • 45,752
  • 12
  • 65
  • 86
  • 1
    That's exactly what I was hoping not to have to do - but thanks for the answer. Maybe one day I'll get round to writing that amazing python distribution package myself... – danodonovan Jan 17 '13 at 10:27
  • 1
    @danodonovan I was similarly disappointed when I wanted this information and found I had to write the above function. It seems like something that should be easily accessible through distutils. At least this way you can figure out the path programatically and don't have to hard-code it. – SethMMorton Jan 17 '13 at 16:35
  • 2
    In shorter form: python -c 'import sys,sysconfig; print("build/lib.{}-{}.{}".format(sysconfig.get_platform(), *sys.version_info[:2]))' – zbyszek Jul 08 '15 at 18:40
  • Note that on Python 3, Python modules go in just lib, so if your test scripts need access to that as well, you'll have to add this without the version info. – jtniehof Oct 30 '15 at 17:31
  • 1
    This is now invalid as of setuptools 62.1.0! Please do not use it. The build directory changed to fix clashes between PyPy and CPython (or any other alternative implementation). It now uses the build tag instead of `{platform}-{version[0]}.{version[1]}`. – Henry Schreiner Apr 12 '22 at 14:30
  • @HenrySchreiner Please edit this answer to state this please. – SethMMorton Apr 12 '22 at 14:51
6

This is available as the build_lib member of the build command class, and it's relative to where setup.py would be. Getting at it when you're not actually running setup is rather nasty:

import distutils.dist
import distutils.command.build
b = distutils.command.build.build(distutils.dist.Distribution())
b.finalize_options()
print(b.build_lib)

Personally I'd go with SethMMorton's solution, since although it's potentially fragile if distutils changes the build directory location, I suspect playing these kinds of tricks is even more fragile if distutils structure changes.

jtniehof
  • 581
  • 3
  • 8
  • 2
    A word of warning: `distutils.dist.Distribution()` won't be enough if, for instance, your `setup.py` builds a native extension. In that case, you might need to create a `Distribution` from your `setup.py`'s metadata dict. – zneak Sep 18 '17 at 20:02
  • You were correct, the build directory just changed in setuptools 62.1, fixing long standing bugs with PyPy + CPython, but invalidating the manual construction. – Henry Schreiner Apr 12 '22 at 14:29
5

I found that compiling the module in-place is best for testing purposes. To do so just use 

python setup.py build_ext --inplace

This will compile all the auxiliary files in the temp directory as usual, but the final .so file will be put in the current directory.

Dejan Jovanović
  • 2,085
  • 1
  • 16
  • 22
0

@SethMMorton's approach is the way to go. It might be however safer (even if I don't know a platform for which his version doesn't work) to use get_platform() from distutils.util rather than from sysconfig.

This is how it is used in distutils-library:

import os
import sys
from distutils.util import get_platform
def get_distutils_lib_path():
    PLAT_SPEC = "%s-%d.%d" % (get_platform(), *sys.version_info[:2])
    return os.path.join("build", "lib.%s" % PLAT_SPEC)
ead
  • 32,758
  • 6
  • 90
  • 153