python setup.py sdist and custom setup keywords don't play together

Question

subtitle: Not only sdist

I am trying to get the setup.py file of a package I'm working on to play nicely with sdist. The relevant parts of the setup.py file are:

from setuptools.command.test import test
[...]
class Tox(test):
   "as described in 
    http://tox.readthedocs.org/en/latest/example/basic.html?highlight=setuptools#integration-with-setuptools-distribute-test-commands"
   [...]
def run_tests(self):
    if self.distribution.install_requires:
        self.distribution.fetch_build_eggs(
            self.distribution.install_requires)
    if self.distribution.tox_requires:
        self.distribution.fetch_build_eggs(self.distribution.tox_requires)
    # import here, cause outside the eggs aren't loaded
    import tox
    import shlex
    args = self.tox_args
    if args:
        args = shlex.split(self.tox_args)
    else:
        args = ""
    errno = tox.cmdline(args=args)
    sys.exit(errno)


entry_points ={}
distutils_ext = {'distutils.setup_keywords': [
                    "tox_requires = setuptools.dist:check_requirements", ]
                 }
entry_points.update(distutils_ext)

setup(
      install_requires=['six', 'numpy', 'matplotlib', 'scipy', 'astropy>=1',
                  'Pillow', ],

    cmdclass={
        'test': PyTest,  # this is to run python setup.py test
        'tox': Tox,
    },

    # list of packages and data
    packages=find_packages(),

    # tests
    tests_require=['pytest', 'pytest-cov'],
    tox_requires=['tox'],
    # other keywords, mostly metadata
)

If I run python setup.py sdist, I get a warning at the beginning:

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'tox_requires'
  warnings.warn(msg)

but then the sdist works fine and it creates a tar.gz file that I can use to install my package.

But if I run it a second time, it starts with (it's the beginning of Pillow building):

warning: no previously-included files found matching '.editorconfig'
Building using 4 processes
_imaging.c: In function ‘getink’:

and begins building all the required packages into .eggs directory.

If I remove the *egg-info directory I can rerun the command. If I comment out the tox_requires=[...] line, I can build the sdist as many times as I want.

Now according to the setuptools documentation the command above should be the correct way to run add new arguments to the setup function.

---

As per the subtitle, the problem is not only with sdist but it's probably due to my non-understanding on how setuptools and requirements work.

If I run python setup.py tox in a place without tox already installed I get, after the installation of some testing package it should not install (namely pytest and pytest-cov):

Traceback (most recent call last): [...] File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 127, in with_project_on_sys_path func() File "setup.py", line 65, in run_tests if self.distribution.tox_requires: AttributeError: Distribution instance has no attribute 'tox_requires'

---

[Update] The tox_requires also confuse very badly pip during installation. If it is commented out I can install the package without any issue; otherwise it begins to compile the source of the packages and it systematically fails because it doesn't find numpy while building stuff like scipy

---

How can I get setuptools to recognize and properly use tox_requires?

Once this issue is fixed, I think that I can get rid of the spurious installations here doing a better job at implementing the Tox class, maybe overriding more things from test or deriving it directly from Command

Answer 1

Complete (working) solution described below consist of 8 files (incl. short README.rst) and has in total 43 lines of code. This is less then code in your original question.

Despite of being so short, it supports many development and testing scenarios in very convenient way.

Anyway, it does not answer exactly your question, but I am sure, it fulfils the requirements which were behind it.

Three lines long setup.py

Technically it may be possible to put test command including tox automation into your setup.py, however, the result may be very messy and difficult to understand.

The same result can be achieved in much simpler way:

• for developer assume:

  • using git
  • having tox installed into system

• for package user:

  • there are no special requirements to install the resulting package

• (optional) if you want your users to test the package by single command and keep test reports collected in central server:

  • install devpi-server and give your users access to it
  • ask your users to install $ pip install devpi

The solution builds on following tools and packages:

• pbr: simplify package creation incl. versioning via git tags and creation of AUTHORS and ChangeLog from git commit messages.
• pytest: excelent testing framework, but any other framework can be used instead of it.
• tox: excellent build and test automation tool.
• coverage: tools to measure test coverage (working simpler then pytest-cov)

Optionally you may also use:

• devpi-server: private PyPi server with password protected access. Allows simple testing and provides test reports collection.
• devpi: tool similar to pip. Apart from installation also supports running tox defined tests (install, run tests, publish reports in on step).

Authoring the package

Create new project directory and initialize git:

$ mkdir francesco
$ cd francesco
$ git init

Create a package or module

Here we create single module francesco, but the same works for more modules or packages.

francesco.py

def main():
    print("Hi, it is me, Francesco, keeping things simple.")

requirements.txt

Create list of packages for actual installation of the package:

six

test_requirements.txt

Define packages required for testing:

pytest
coverage

tests/test_it.py

Initiate the test suite:

from francesco import main


def test_this():
    main()
    print("All seems fine to me")
    assert True

setup.py

Did you dream of stupid simple setup.py? Here it goes:

from setuptools import setup

setup(setup_requires=["pbr"], pbr=True)

setup.cfg

Metadata belong to configuration file:

[metadata]
name = francesco
author = Francesco Montesano
author-email = fm@acme.com
summary = Nice and simply installed python module supporting testing in different pythons
description-file = README.rst
[files]
modules=francesco
[entry_points]
console_scripts =
    francesco = francesco:main

tox.ini

To configure tox automated builds and tests:

[tox]
envlist = py27, py34

[testenv]
commands =
    coverage run --source francesco -m pytest -sv tests
    coverage report
    coverage html
deps =
    -rtest_requirements.txt

README.rst

Never forget README.rst:

===========================================
Complex package with 3 line long `setup.py`
===========================================

Can we keep`setup.py` simple and still support automated testing?

...

tox: build sdist and run tests in all supported python versions

Being in the project directory root, just run single command tox:

$ tox
GLOB sdist-make: /home/javl/sandbox/setuppy/setup.py
py27 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip
py27 runtests: PYTHONHASHSEED='2409409075'
py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7
cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile: 
collecting ... collected 1 items

tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED

=========================== 1 passed in 0.01 seconds ===========================
py27 runtests: commands[1] | coverage report
Name           Stmts   Miss  Cover
----------------------------------
francesco.py       2      0   100%
py27 runtests: commands[2] | coverage html
py34 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip
py34 runtests: PYTHONHASHSEED='2409409075'
py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux -- Python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py34/bin/python3.4
cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile: 
collecting ... collected 1 items

tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED

=========================== 1 passed in 0.01 seconds ===========================
py34 runtests: commands[1] | coverage report
Name           Stmts   Miss  Cover
----------------------------------
francesco.py       2      0   100%
py34 runtests: commands[2] | coverage html
___________________________________ summary ____________________________________
  py27: commands succeeded
  py34: commands succeeded
  congratulations :)

Getting the sdist

ls .tox/dist
francesco-0.0.0.zip

Developing in Python 2.7 virtualenv

Activate Python 2.7 virtualenv

$ source .tox/py27/bin/activate

Run tests

(py27) $ py.test -sv tests

==============================================================================================
test session starts
===============================================================================================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
-- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7 cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile: collected 1 items

tests/test_it.py::test_this Hi, it is me, Francesco, keeping things
simple. All seems fine to me PASSED

============================================================================================
1 passed in 0.01 seconds
============================================================================================

Measure test coverage

(py27)$ coverage run --source francesco -m pytest -sv tests
.....
(py27)$ coverage report
Name           Stmts   Miss  Cover
----------------------------------
francesco.py       2      0   100%

View coverage report in web browser

(py27)$ coverage html
(py27)$ firefox htmlcov/index.html

Release new package version

(optional) Install local devpi-server

The installation of devpi-server is not covered here, but is very simple, especially, if you install only to your local machine for your personal testing.

Commit source code, assign version tag

Make sure, all your source code is commited.

Assing version tag:

$ git tag -a 0.1

Rerun the tests by tox and build sdist

Make sure, you have deactivated virtualenv (otherwise it conflicts with tox):

(py27)$ deactivate

Run the tox:

$ tox
.....
...it builds as usual, may fail, if you have forgotten to commit some changes or files...

Find the sdist for new version of your package:

$ ls .tox/dist/francesco-0.1.0.
.tox/dist/francesco-0.1.0.zip

You are done. You may distribute your new tested versions of your package to users as usually.

(optional) Upload the sdist to devpi-server and test it locally

Following steps assume, you have devpi-server installed and running.

$ devpi login javl
...enter your password...
$ devpi upload .tox/dist/francesco-0.1.0.zip

Test the package in clean environment

(deactivate virtualenv if active) :

$ cd /tmp
$ mkdir testing
$ cd testing
$ devpi test francesco
received http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip
unpacking /tmp/devpi-test6/downloads/francesco-0.1.0.zip to /tmp/devpi-test6/zip
/tmp/devpi-test6/zip/francesco-0.1.0$ tox --installpkg /tmp/devpi-test6/downloads/francesco-0.1.0.zip -i ALL=http://localhost:3141/javl/dev/+simple/ --recreate --result-json /tmp/devpi-test6/zip/toxreport.json
-c /tmp/devpi-test6/zip/francesco-0.1.0/tox.ini
py27 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27
py27 installdeps: -rtest_requirements.txt
py27 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip
py27 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='3916044270'
py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27/bin/python2.7
cachedir: .cache
rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile:
collecting ... collected 1 items

tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED

=========================== 1 passed in 0.01 seconds ===========================
py27 runtests: commands[1] | coverage report
Name           Stmts   Miss  Cover
----------------------------------
francesco.py       2      0   100%
py27 runtests: commands[2] | coverage html
py34 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34
py34 installdeps: -rtest_requirements.txt
py34 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip
py34 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0
py34 runtests: PYTHONHASHSEED='3916044270'
py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux -- Python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34/bin/python3.4
cachedir: .cache
rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile:
collecting ... collected 1 items

tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED

=========================== 1 passed in 0.01 seconds ===========================
py34 runtests: commands[1] | coverage report
Name           Stmts   Miss  Cover
----------------------------------
francesco.py       2      0   100%
py34 runtests: commands[2] | coverage html
____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
  py27: commands succeeded
  py34: commands succeeded
  congratulations :)
wrote json report at: /tmp/devpi-test6/zip/toxreport.json
posting tox result data to http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip
successfully posted tox result data

You may check the test results in web browser:

$ firefox http://localhost:3141

then search for "francesco" package, click the package name, find in table column named "tox results", click there to show environment set up and test results.

Let your users test the package

Let's assume, your devpi-server is running and your user has access to it.

The user shall have devpi command installed:

$ pip install devpi

(note, this tool is not installing anything from the devpi-server)

Help your user to gain access to devpi-server (not covering here).

Then the user just runs the test:

$ devpi test francesco

After the test is run (it is automatically using tox, but user does not have to care about that), you will find test results on the same place on devpi web interface as you found yours before.