8

I am developing a package containing Cython extensions.

According to https://github.com/pypa/pip/issues/1958 I shall use setup_requires and postpone import of Cython. The best solution I came up with is to call setup() twice in setup.py:

... # initial imports
setup(setup_requires=['cython'])
from Cython.Build import cythonize
bar = Extension('foo.bar', sources = ['bar.pyx'])
setup(name = 'foo',
      ... # parameters
      ext_modules = cythonize([bar]),
      ... # more parameters
      )

However I have a feeling that the name of setup() suggests it shall be called only once. Is it safe to call it several times like I do?

I can not just distribute wheels because the package shall be available also for Linux users.

[EDIT]

Also I see the question as more general than dealing with compiler-dependencies. One may want to import some package (eg. sphinx or pweave) to preprocess the description of ones package.

abukaj
  • 2,582
  • 1
  • 22
  • 45

2 Answers2

8

I had a different scenario where I needed to run setup() multiple times: in my case I was building two packages from the same sources. First package was a command line tool based on Fabric, second package was only library (APIs, tooling, etc.) It seemed too impractical to split the project to two repositories for such a small project, since the CLI part was really only a wrapper. Running setup() multiple times with different arguments caused build crashing on various errors (mostly missing files). My solution was to run each setup() as different Process:

from setuptools import setup, find_packages
from multiprocessing import Process

if __name__ == '__main__':
    setups = [
        {
            'name': 'cli-tool',
            'description': 'Some description...',
            'packages': find_packages(),
            'entry_points': {
                'console_scripts': [
                    'cli-tool = fabfile:main'
                ]
            },
            '...': 'etc. needed for setup() ...'
        },
        {
            'name': 'cli-tool-lib',
            'packages': find_packages(exclude=('fabfile',)),
            '...': 'etc.'
        }
    ]

    for s in setups:
        name = s['name']
        print("Building '{}'.".format(name))
        p = Process(target=setup, kwargs=s)
        p.start()
        p.join()
        print("Building of '{}' done.\n".format(name))
David Lukac
  • 728
  • 7
  • 20
  • It doesn't work with `pip install .`. My solution is to have two setup.py files in different directories. So user have to install via `pip install ./subdirectory1 ./subdirectory2` (and for git+ urls use `#egg=...&subdirectory=...`). – ArtemSBulgakov Jul 12 '20 at 20:48
  • I prefer this solution: https://stackoverflow.com/a/770247/1763602 – Marco Sulla Oct 22 '20 at 17:45
4

The simple answer is: No. Once you call setup, it will parse the command line arguments and start doing its job.

As for Cython dependency, setup_requires cannot help here. It will probably try to download Cython without installing. As SpotlightKid commented:

distutils doesn't try to be a compiler or install gcc as a dependency either

According to the setuptools

this argument (setup_requires) is needed if you are using distutils extensions,

and thus, not for packages like Cython.

I think the user is responsible to install Cython before calling setup.py. If you want to provide more friendly error message, try to use

try:
    from Cython.Build import cythonize
except ImportError:
    # Kindly ask the user to install Cython

The following posts may help:

gdlmx
  • 6,479
  • 1
  • 21
  • 39
  • "it will parse the command line arguments and start doing its job" - it is exactly what made me doubt, but is it explicitly mentioned somewhere in the docs? Someone may wanted to inject a non-compiler dependency in a similar pattern. – abukaj Sep 08 '17 at 17:07
  • @abukaj It is not mentioned in the docs, but you can easily find it out in the [source](https://bitbucket.org/carljm/python-distutils/src/48c42eeaee4410d76675b637bcd401b8919ff19a/core.py?at=default&fileviewer=file-view-default#core.py-59). More important is: even if it is safe to do so, the setuptools doesn't really help in this situation. – gdlmx Sep 08 '17 at 19:12