3

I've got a package which I've previously successfully built on ReadTheDocs, but this is no longer the case. My imports are as follows:

import pandas as pd
import numpy as np
import scipy
import sys
from annoy import AnnoyIndex
from packaging import version
from scipy.spatial import cKDTree
from scipy.sparse import coo_matrix
from umap.umap_ import fuzzy_simplicial_set
from sklearn.neighbors import KDTree
from sklearn.neighbors import DistanceMetric
from sklearn.linear_model import Ridge

I cover these packages via mock importing in Sphinx.

autodoc_mock_imports = ['pandas','numpy','scipy','annoy','packaging','scipy.spatial','scipy.sparse','sklearn','sklearn.neighbors','sklearn.linear_model','umap','umap.umap_']

However, in spite of that, I appear to encounter issues when building ReadTheDocs side. It seems that despite being asked to mock sklearn up, the thing's trying to install it and failing.

Installed /home/docs/checkouts/readthedocs.org/user_builds/bbknn/envs/latest/lib/python3.7/site-packages/bbknn-1.4.0-py3.7.egg
Processing dependencies for bbknn==1.4.0
Searching for scikit-learn
Reading https://pypi.org/simple/scikit-learn/
Downloading https://files.pythonhosted.org/packages/db/e2/9c0bde5f81394b627f623557690536b12017b84988a4a1f98ec826edab9e/scikit-learn-0.24.0.tar.gz#sha256=076369634ee72b5a5941440661e2f306ff4ac30903802dc52031c7e9199ac640
Best match: scikit-learn 0.24.0
Processing scikit-learn-0.24.0.tar.gz
Writing /tmp/easy_install-1f85jnk0/scikit-learn-0.24.0/setup.cfg
Running scikit-learn-0.24.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-1f85jnk0/scikit-learn-0.24.0/egg-dist-tmp-mfl2n2vi
Partial import of sklearn during the build process.
Traceback (most recent call last):
  File "/tmp/easy_install-1f85jnk0/scikit-learn-0.24.0/setup.py", line 201, in check_package_status
    module = importlib.import_module(package)
  File "/home/docs/checkouts/readthedocs.org/user_builds/bbknn/envs/latest/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'numpy'
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/bbknn/envs/latest/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/home/docs/checkouts/readthedocs.org/user_builds/bbknn/envs/latest/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/home/docs/checkouts/readthedocs.org/user_builds/bbknn/envs/latest/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/home/docs/checkouts/readthedocs.org/user_builds/bbknn/envs/latest/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-1f85jnk0/scikit-learn-0.24.0/setup.py", line 306, in <module>
  File "/tmp/easy_install-1f85jnk0/scikit-learn-0.24.0/setup.py", line 292, in setup_package
  File "/tmp/easy_install-1f85jnk0/scikit-learn-0.24.0/setup.py", line 227, in check_package_status
ImportError: numpy is not installed.
scikit-learn requires numpy >= 1.13.3.
Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html

Why is this happening? Could this be related to the way I require dependencies in setup.py, which somehow circumvents the mocks?

install_requires=['Cython','numpy','scipy','annoy','umap-learn','scikit-learn','packaging'],

I have hotfix-generated a docstring by removing umap-learn and scikit-learn from install_requires, but that's not an ideal solution. Anybody got anything?

ktpolanski
  • 31
  • 3

1 Answers1

0

Based on feedback from a helpful user, I eventually arrived at a less hack'y solution. Unfortunately, the discussion went AWOL because of an unhelpful user, who responded with pip install -U numpy, waited for me to figure it out, edited their answer and requested I accept it. Upon being denied, the answer and comment thread vanished. I don't even remember your name, helpful user, so I can't credit you for the tip.

Apparently ReadTheDocs uses an old pip, and requiring pip>=19.0 makes scikit-learn not install from source. As such, I added that line to docs/requirements.txt, which I had previously set up to be a ReadTheDocs requirement file. This resulted in some progress - now rather than scikit-learn complaining about numpy, it was numba. Still, some synapses connected, and I just handled any dependency problems that arose via docs/requirements.txt, the final contents of which are:

pip>=19.0
numpy
llvmlite==0.35.0

This got the setup to clear. However, just because setup cleared, does not mean that autodoc_mock_imports can be stripped away for whatever reason. Not sure why this is.

As such, the quick and dirty solution from the OP wherein you just strip stuff out of setup.py's install_requires is likely the easiest way around this. However, if you feel like faffing around to get it to be cleaner, and not requiring a dedicated non-functional commit to have documentation build, having a ReadTheDocs requirement file and installing numpy through it is likely to solve a lot of woes. No idea why llvmlite was also this fussy, and somehow worked when in the requirements file but not otherwise, but got there in the end.

ktpolanski
  • 31
  • 3