28

working on a python project, I tried to separate source code and unit tests; here is the project structure:

MyProject/
    MANIFEST.in
    README.md
    setup.py
    source/
        __init.py__
        my_project/
            __init.py__
            some_module.py
    test/
        __init.py__
        my_project/
            __init.py__
            test_some_module.py

And here is the setup.py file:

from setuptools import setup, find_packages

setup(
    name='my_project',
    packages=find_packages(where='./source'),
    description='My project to be packaged',
    version='1.0.0',
    author='me'
    install_requires=[
        'fastnumbers~=2.0.1',
        'numpy~=1.14.1',
        'pandas~=0.22.0'
    ],
    extras_require={
        'dev': ['check-manifest'],
        'test': [
            'mock',
            'PyHamcrest',
            'pytest',
            'pytest-cov'
        ],
    }
)

Then, when I run command python3 setup.py sdist, it fails with the following output:

running sdist
running egg_info
writing my_project.egg-info/PKG-INFO
writing requirements to my_project.egg-info/requires.txt
writing dependency_links to my_project.egg-info/dependency_links.txt
writing top-level names to my_project.egg-info/top_level.txt
error: package directory 'my_project' does not exist

The resulting top_level.txt file looks fine:

 my_project

but it looks like the setuptools is not starting from source folder to find modules to be packaged.

  1. Do I have to move setup.py and MANIFEST.in files into source folder?
  2. But then, what is this where argument for in setuptools.find_packages function?
Géraud
  • 1,923
  • 3
  • 20
  • 20

1 Answers1

33

You are one step away from a working solution. Add

package_dir={
    '': 'source',
},

to the setup() arguments:

setup(
    ...,
    packages=find_packages(where='source'),
    package_dir={
        '': 'source',
    },
    ...
)

More info on packages remapping can be found in Listing whole packages section.

However, it looks like you made the source dir to a python package by placing an __init__.py in it. Was that intentional? Do you have import statements like

import source.my_project
from source.my_project.my_module import stuff

or similar, using source as package name? Then beware that the imports will fail once you install your built package because source is omitted when including sources on build. I see two ways:

  1. either remove source/__init__.py, use package_dir like described above to make my_project to the top level package, omit source in imports (should you get any errors, just remove myproject-1.0.0.egg_info dir and recreate it with python setup.py egg_info), or
  2. use source as top level package: don't use package_dir, look for packages in the project root dir (packages=find_packages() without explicitly stating where).
hoefling
  • 59,418
  • 12
  • 147
  • 194
  • 2
    Thank you @hoefling! This `package_dir` option solved my issue. And good catch also about the unnecessary `__init__.py` file! – Géraud Jul 11 '18 at 22:11
  • Glad I could help! – hoefling Jul 11 '18 at 22:17
  • 2
    Funny how a solution to someone else's problems goes a long way to helping others! Thank @hoefling for this tip, it has helped me today. – Helen Neely Mar 13 '19 at 12:30
  • @HelenNeely glad to hear that! Should you need more information for the sources remapping on packaging, check out the [Listing whole packages](https://docs.python.org/3/distutils/setupscript.html#listing-whole-packages) section in the stdlib docs. – hoefling Mar 13 '19 at 19:39
  • 1
    I just found an helpful article about python libraries packaging, and it might help other users: https://blog.ionelmc.ro/2014/05/25/python-packaging – Géraud Oct 18 '19 at 13:17