5

Note: Since asking this question, I discovered later on that python -m pip install -e . will install the extension to cmod with .venv/lib/python3.8/site-packages/hello-c-extension.egg-link pointing to the project in the current directory. I've also switched to a src layout in later commits and have found https://pythonwheels.com/ to be a great reference for high-quality packages that distribute wheels. However, I'm still curious to know about the behavior of setup.py subcommands.

As part of some research on manylinux, I am using a toy project to build different platform wheels for a C++ extension module.

It seems that when building and installing locally, I cannot import the C++ extension module if my current directory is the project root directory. This prevents me from running unit tests, among other things. I believe the reason for this is that . becomes the first component of sys.path, and so the pure-Python version is picked up while the compiled extension is not.

How can I fix this? Am I running the local build/install correctly?

The package structure looks like this:

$ tree hello-c-extension/
hello-c-extension/
├── LICENSE
├── Makefile
├── README.md
├── cmod
│   ├── __init__.py
│   ├── _cmodule.cc
│   └── pymod.py
├── setup.py
└── tests
    ├── __init__.py
    └── test_cext.py

I also have the project on GitHub; I asked this question as of 29fef5b.

To build/install I use:

cd hello-c-extension
python -m venv .venv
source ./.venv/bin/activate
python -m pip install -U pip wheel setuptools
python setup.py build install

Now, from the current directory, I can import the Python module but not the corresponding extension module. The Python module gets picked up as the one in the current directory, rather than that in site-packages:

$ python -c 'from cmod import pymod; print(pymod)'
<module 'cmod.pymod' from '/Users/brad/Scripts/python/projects/bsolomon1124/hello-c-extension/cmod/pymod.py'>
$ python -c 'from cmod import _cmod; print(_cmod)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name '_cmod' from 'cmod' (/Users/brad/Scripts/python/projects/bsolomon1124/hello-c-extension/cmod/__init__.py)

Hackishly deleting the PWD element of sys.path fixes this:

>>> import sys
>>> sys.path
['', '/Users/brad/.pyenv/versions/3.8.1/lib/python38.zip', '/Users/brad/.pyenv/versions/3.8.1/lib/python3.8', '/Users/brad/.pyenv/versions/3.8.1/lib/python3.8/lib-dynload', '/Users/brad/Scripts/python/projects/bsolomon1124/hello-c-extension/.venv/lib/python3.8/site-packages', '/Users/brad/Scripts/python/projects/bsolomon1124/hello-c-extension/.venv/lib/python3.8/site-packages/hello_c_extension-0.4-py3.8-macosx-10.15-x86_64.egg']
>>> del sys.path[0]
>>> from cmod import _cmod; print(_cmod)
<module 'cmod._cmod' from '/Users/brad/Scripts/python/projects/bsolomon1124/hello-c-extension/.venv/lib/python3.8/site-packages/hello_c_extension-0.4-py3.8-macosx-10.15-x86_64.egg/cmod/_cmod.cpython-38-darwin.so'>

And finally, changing out of the directory makes the problem go away as well:

$ cd ..
$ python -c 'from cmod import _cmod; print(_cmod)'
<module 'cmod._cmod' from '/Users/brad/Scripts/python/projects/bsolomon1124/hello-c-extension/.venv/lib/python3.8/site-packages/hello_c_extension-0.4-py3.8-macosx-10.15-x86_64.egg/cmod/_cmod.cpython-38-darwin.so'>

Is this really ... how it's supposed to work? What would be the proper way to run unit tests for the extension module in this case?

System info:

$ python -V
Python 3.8.1
$ uname -mrsv
Darwin 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64
Brad Solomon
  • 38,521
  • 31
  • 149
  • 235
  • It is recommended to put your module dir in a subdirectory and not straight at the root of the package, precisely to avoid this problem, and to force using (any) installed version. – Tronic Apr 05 '20 at 12:40
  • With the alternative being to use a [`src` layout](https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure) @L.Kärkkäinen? – Brad Solomon Apr 05 '20 at 12:43
  • Every module I've looked at seems to be using something different. `src/*.c` and `module/name/*.py` and other variations. Dunno about the build parts, that's why I commented and didn't answer. – Tronic Apr 05 '20 at 12:46
  • 1
    The ['alternatives'](https://blog.ionelmc.ro/2014/05/25/python-packaging/#alternatives) section of the article I linked above seems to mention this issue specifically. (While [this older article](https://as.ynchrono.us/2007/12/filesystem-structure-of-python-project_21.html) cautions, ironically, "Don't put your source in a directory called src or lib. This makes it hard to run without installing.") – Brad Solomon Apr 05 '20 at 12:51
  • 1
    `pip install -e .` makes it easy enough to install, so that should no longer be a concern. – Tronic Apr 05 '20 at 13:59
  • 1
    Works fine with *Python 3.5* (*Ubtu 16*). – CristiFati Apr 06 '20 at 21:48

1 Answers1

2

The cause of the problems

Is this really ... how it's supposed to work? What would be the proper way to run unit tests for the extension module in this case?

Python imports from sys.path. This means that it will search each and every directory for the import, from index 0 through to the last item in sys.path.

Note that a hacky approach to a fix would be to add the following before the import: sys.path.append(sys.path.pop(0)). This still allows local imports, but means that site-packages and the standard library are searched first.

How can I fix this? Am I running the local build/install correctly?

I'll answer the first question below.

Yes, your build/install is fine, but to fix the import problem, you'll need to tweak it slightly.

Fixing the problem

Switch to the src/cmod layout. A great link - which is pointed out by yourself in the questions comments - is here.

This approach is discussed in the comments, and since asking the question you've actually implemented this. (60093f1).

I'll now quote some of that article; it explains this better than I could.

The src directory is a better approach because:

  • You get import parity. The current directory is implicitly included in sys.path; but not so when installing & importing from site-packages. Users will never have the same current working directory as you do.

This constraint has beneficial implications in both testing and packaging:

  • You will be forced to test the installed code (e.g.: by installing in a virtualenv). This will ensure that the deployed code works (it's packaged correctly) - otherwise your tests will fail. Early. Before you can publish a broken distribution.

  • You will be forced to install the distribution. If you ever uploaded a distribution on PyPI with missing modules or broken dependencies it's because you didn't test the installation. Just beeing able to successfuly build the sdist doesn't guarantee it will actually install!

And pip install -e . will no longer mess stuff up.

Legorooj
  • 2,646
  • 2
  • 15
  • 35