Installing data_files in setup.py with pip install -e

Question

I'm trying to provide a bash completion script for my CLI tool that is written in Python. According to the Python Packaging Authority, data_files in setup.py is exactly what I need:

Although configuring package_data is sufficient for most needs, in some cases you may need to place data files outside of your packages. The data_files directive allows you to do that. It is mostly useful if you need to install files which are used by other programs, which may be unaware of Python packages.

So I added the completion file like this:

data_files=[
    ('/usr/share/bash-completion/completions', ['completion/dotenv']),
],

and try to test it with:

pip install -e .

In my virtual environment. However, the completion script gets not installed. Did I forgot something or is pip broken? The full project can be found here

Just a thought...did you check your output carefully? Might there be some sort of error or warning message there? — CryptoFool, Mar 17 '19 at 14:58
Nothing, not even a warning or anything, even when I install with `-vvv`. — Bastian Venthur, Mar 17 '19 at 15:04
To install into `/usr` your users must run `pip install` as root. I.e., you forbid to use `pip install --user` or installing into a virtual environment. The second problem is that `bash` stores completion in `/usr/share/bash-completion/completions` only on Linux. I.e., the package installs completion file into a wrong directory at FreeBSD or MacOS. In short, `data_files` is is a completely wrong approach. — phd, Mar 17 '19 at 16:59
See also https://docs.python.org/2/distutils/setupscript.html#installing-additional-files: "**The directory should be a relative path**. It is interpreted relative to the installation prefix (Python’s sys.prefix for system installations; site.USER_BASE for user installations). Distutils allows directory to be an **absolute installation path, but this is discouraged** since it is incompatible with the wheel packaging format. No directory information from files is used to determine the final location of the installed file; only the name of the file is used." — phd, Mar 17 '19 at 17:06

Samir Sadek · Answer 1 · 2020-03-05T02:49:24.297

I had the same issue and I have implemented a workaround.

It seems to me that python setup.py develop or (pip install -e .) does not run the same function than python setup.py install. In fact, I have noticed by looking in the source code that python setup.py install run build_py :

https://github.com/python/cpython/blob/master/Lib/distutils/command/build_py.py#L134 https://github.com/pypa/setuptools/blob/master/setuptools/command/build_py.py

After a few digging I have opted to override the develop command as follow. The following code is python3.6:

""" SetupTool Entry Point """
import sys
from pathlib import Path
from shutil import copy2

from setuptools import find_packages, setup
from setuptools.command.develop import develop

# create os_data_files that will be used by the default install command
os_data_files = [
    (
        f"{sys.prefix}/config",  # providing absolute path, sys.prefix will be different in venv
        [
            "src/my_package/config/properties.env",
        ],
    ),        
]


def build_package_data():
    """ implement the necessary function for develop """
    for dest_dir, filenames in os_data_files:
        for filename in filenames:
            print(
                "CUSTOM SETUP.PY (build_package_data): copy %s to %s"
                % (filename, dest_dir)
            )
            copy2(filename, dest_dir)


def make_dirstruct():
    """ Set the the logging path """
    for subdir in ["config"]:
        print("CUSTOM SETUP.PY (make_dirstruct): creating %s" % subdir)
        (Path(BASE_DIR) / subdir).mkdir(parents=True, exist_ok=True)


class CustomDevelopCommand(develop):
    """ Customized setuptools install command """

    def run(self):
        develop.run(self)
        make_dirstruct()
        build_package_data()

# provide the relevant information for stackoverflow
setup(        
    package_dir={"": "src"},
    packages=find_packages("src"),
    data_files=os_data_files,                
    cmdclass={"develop": CustomDevelopCommand},
)

Installing data_files in setup.py with pip install -e

1 Answers1