36

I'm trying to create a setup.py file where find_packages() recursively finds packages. In this example, foo, bar, and baz are all modules that I want to be installed and available on the python path. For example, I want to be able to do import foo, bar, baz. The bar-pack and foo-pack are just regular non-python directories that will contain various support files/dirs (such as tests, READMEs, etc. specific to the respective module). 

├── bar-pack
│   └── bar
│       └── __init__.py
├── baz
│   └── __init__.py
├── foo-pack
│   └── foo
│       └── __init__.py
├── setup.py

Then say that setup.py is as follows:

from setuptools import setup, find_packages
setup(
    name="mypackage",
    version="0.1",
    packages=find_packages(),
)

However, when I run python setup.py install or python setup.py sdist, only the baz directory is identified and packaged.

I can simplify it down further, and run the following command, but again, only baz is identified. 

python -c "from setuptools import setup, find_packages; print(find_packages())"
['baz']

Do you know how I might extend the search path (or manually hard-code the search path) of the find_packages()?

Any help is appreciated.

Joe J
  • 9,985
  • 16
  • 68
  • 100
  • 2
    Interestingly for me, folders that were not packaged into wheel are the folders that do not have `__init__.py` file. As soon as I placed the empty `__init__.py` file in the folder which needs to go into wheel, folders/files were rolled into wheel package. – Sunil Apr 01 '22 at 04:54

2 Answers2

43

This is like using the src-layout for the "foo" and "bar" packages, but the flat layout for "baz". It's possible, but requires some custom configuration in the setup.py.

Setuptools' find_packages supports a "where" keyword (docs), you can use that.

setup(
    ...
    packages=(
        find_packages() +
        find_packages(where="./bar-pack") +
        find_packages(where="./foo-pack")
    ),
    ...
)

Since find_packages returns a plain old list, you could also just list your packages manually, and that's arguably easier / less magical.

setup(
    ...
    packages=["baz", "bar", "foo"],
    ...
)

The non-standard directory structure means you'll also want to specify the package_dir structure for distutils, which describes where to put the installed package(s).

Piecing it all together:

setup(
    name="mypackage",
    version="0.1",
    packages=["baz", "bar", "foo"],
    package_dir={
        "": ".",
        "bar": "./bar-pack/bar",
        "foo": "./foo-pack/foo",
    },
)

The above installer will create this directory structure in site-packages:

.venv/lib/python3.9/site-packages
├── bar
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-39.pyc
├── baz
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-39.pyc
├── foo
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-39.pyc
└── mypackage-0.1.dist-info
    ├── INSTALLER
    ├── METADATA
    ├── RECORD
    ├── REQUESTED
    ├── WHEEL
    ├── direct_url.json
    └── top_level.txt
wim
  • 338,267
  • 99
  • 616
  • 750
  • 1
    Is it safe to say that the `where` and the `package_dir` pieces are both needed? When I tried specifying `find_packages() + find_packages(where="./bar-pack")` without the `package_dir` structure, pip coughed up an error `error: package directory 'bar-pack' does not exist`. I think you're right that just specifying the package names directly (avoiding `find_packages`) yields a more understandable solution. – chrisinmtown Apr 22 '22 at 15:56
  • 2
    @chrisinmtown Yes, I think they are both needed. One advantage of `find_packages` is that if you list packages manually it's easy to forget you need to add sub-packages - e.g. if there was a `bar/subbar/__init__.py`, you'd have to list both "bar" and "bar.subbar" as packages. – wim Apr 22 '22 at 16:58
-1

You can also specify by ['packagea','packagea.*']

by this you wont be needed to specify sub packages as well.

Nitin Arora
  • 1