0

I am attempting to build documentation using Sphinx on ReadTheDocs, for a project that depends on a bunch of pydata packages, some of which depend on C libraries which are not available on ReadTheDocs (namely the python-snappy package, which provides the snappy module, implementing Google's snappy compression algorithm, which we are using in Apache Parquet files).

The documentation suggests that the solution is to mock those python modules so that autodoc can import them, even if the code isn't really there, using the autodoc_mock_imports parameter in conf.py So I've got

autodoc_mock_imports = ['snappy', 'python-snappy']

in my Sphinx conf.py (I think it's really supposed to be just snappy in there, but I wasn't 100% sure and it wasn't working with snappy at first, so I added the package name as well as the module name... and it still didn't work).

However, the documentation build is still attempting to compile snappy and predictably failing, because the C-language headers aren't available. When the build is collecting requirements, it finds python-snappy, I think in the project setup.py and says:

Collecting python-snappy (from pudl==0.1.dev398+g5e075d4)

And subsequently the build fails:

Building wheels for collected packages: pudl, python-snappy
  Building wheel for pudl (setup.py): started
  Building wheel for pudl (setup.py): finished with status 'done'
  Stored in directory: /home/docs/checkouts/readthedocs.org/user_builds/catalyst-cooperative-pudl/.cache/pip/wheels/a4/5f/b6/1f8213aeb5876af1c140b54dce3466b845d989ca1101da875a
  Building wheel for python-snappy (setup.py): started
  Building wheel for python-snappy (setup.py): finished with status 'error'
  ERROR: Complete output from command /home/docs/checkouts/readthedocs.org/user_builds/catalyst-cooperative-pudl/envs/python-packaging/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-z6c1cop3/python-snappy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-r54uwk2d --python-tag cp37:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/snappy
  copying snappy/__init__.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/snappy_cffi_builder.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/__main__.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/hadoop_snappy.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/snappy_formats.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/snappy.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/snappy_cffi.py -> build/lib.linux-x86_64-3.7/snappy
  running build_ext
  building 'snappy._snappy' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/snappy
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/docs/.pyenv/versions/3.7.3/include/python3.7m -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.7/snappy/snappymodule.o
  snappy/snappymodule.cc:31:10: fatal error: snappy-c.h: No such file or directory
   #include <snappy-c.h>
            ^~~~~~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for python-snappy
  Running setup.py clean for python-snappy
Successfully built pudl
Failed to build python-snappy

python-snappy is commented out of the requirements.txt that is included in the documentation build, but of course it is still included in the project setup.py in the install_requires parameter.

Why is the mocking not working? How do I get it to stop trying to build this package that can't succeed? On my local system (which has the ``libsnappy-dev` package installed) the documentation build seems to work fine.

The full output from the failed build can be found here:

https://readthedocs.org/projects/catalyst-cooperative-pudl/builds/9376117/

Zane Selvans
  • 11
  • 3

1 Answers1

1

The problem here was that I had also explicitly told Read The Docs to build and install my pudl package using pip, in the .readthedocs.yml configuration file:

# Set the version of Python and requirements required to build your docs
python:
  version: 3.7
  install:
    - requirements: docs/requirements.txt
    - method: pip
      path: .
  system_packages: true

Which meant that it was trying to create a pudl distribution, by running the setup.py found in the top level of the repository, which predictably failed due to the lack of the snappy C headers.

The solution was to conditionally exclude the python-snappy package from the install_requires parameter in setup.py based on the READTHEDOCS environment variable, leaving the dependency to be satisfied by the mocked module as specified by autodoc_mock_imports in conf.py:

install_requires = [
    'coloredlogs',
    'datapackage',
    'dbfread',
    'goodtables',
    'jupyter',
    'jupyterlab',
    'matplotlib',
    'nbval',
    'networkx>=2.2',
    'numpy',
    'pandas>=0.24',
    'psycopg2',
    'pyarrow>=0.14.0',
    'pyyaml',
    'scikit-learn>=0.20',
    'scipy',
    'sqlalchemy>=1.3',
    'sqlalchemy-postgres-copy',
    'tableschema',
    'timezonefinder',
    'xlsxwriter',
]

# We are installing the PUDL module to build the docs, but the C libraries
# required to build snappy aren't available on RTD, so we need to exclude it
# from the installed dependencies here, and mock it for import in docs/conf.py
# using the autodoc_mock_imports parameter:
if not os.getenv('READTHEDOCS'):
    install_requires.append('python-snappy')

Another potential solution would be to add the pudl package directory to sys.path in the conf.py so that the package is importable, even though it's not installed. However, I am using setuptools_scm to automatically generate a version number for use by the packaging and documentation, and for that to work the pudl package needs to actually be built as a distribution and installed.

Zane Selvans
  • 11
  • 3