Sphinx autodoc gets stuck on random.choice() using mock

Question

This is a continuation of a question I originally asked in this post: random.choice error due to np.linspace and np.logspace

In this question, I have stripped down the problem to the bare minimum and was able to reproduce the problem where Sphinx is hanging up on the random.choice() function.

Here is the Python code in the file randor_test.py; it runs in PyCharm:

import random
import numpy as np

def rand_test(svr_C, svr_gamma):
    """This is test docstring

    #. item one
    #. item two
    """
    ml_params = {'C': random.choice(svr_C), 'gamma': random.choice(svr_gamma)}
    return ml_params

svr_C = list(np.linspace(50, 300, 10))
svr_gamma = list(np.logspace(-4, -2, 3))

rand_result = rand_test(svr_C, svr_gamma)

for i in rand_result:
    print(i, rand_result[i])

I set up the Sphinx directory and followed all instructions in this post: Getting Started with Sphinx...

After running make html I receive the following error:

WARNING: autodoc: failed to import module 'randor_test'; the following exception was raised:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sphinx/ext/autodoc/importer.py", line 32, in import_module
    return importlib.import_module(modname)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/kellihed/PAC_projects/basic_0629/src/randor_test.py", line 19, in <module>
    rand_result = rand_test(svr_C, svr_gamma)
  File "/Users/kellihed/PAC_projects/basic_0629/src/randor_test.py", line 10, in rand_test
    ml_params = {'C': random.choice(svr_C), 'gamma': random.choice(svr_gamma)}
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/random.py", line 261, in choice
    raise IndexError('Cannot choose from an empty sequence') from None
IndexError: Cannot choose from an empty sequence

My file structure follows the suggested layout:

|--docs
|  |
|  |--Makefile
|  |--build
|  |  |
|  |  |--doctrees
|  |  |--html
|  |  |--_static
|  |  |--genindex.html
|  |  |--index.html 
|  |  |--modules.html
|  |--make.bat
|  |--source
|     |
|     |--_static
|     |--_templates
|     |--conf.py
|     |--index.rst
|     |--modules.rst
|     |--randor_test.rst
|
|--src
|  |__pychache__
|  |  |
|  |  |--randor_test.cpython-37.pyc
|  |
|  |--randor_test.py

I have the following ignore imports statement in conf.py:

autodoc_mock_imports = ["random", "numpy"]

Below is my conf.py file:

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join('..', '..', 'src')))


# -- Project information -----------------------------------------------------

project = 'random_test'
copyright = '2020, DK'
author = 'DK'

# The full version, including alpha/beta/rc tags
release = '0.1'


# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.autodoc']
autodoc_mock_imports = ["random", "numpy"]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

score 3 · Accepted Answer · answered Jul 01 '20 at 07:28

You have module-level code that will always be executed when Sphinx imports randor_test. svr_C = list(np.linspace(50, 300, 10)) will not cause an error directly, but the result is that svr_C is an an empty list.

The error occurs when random.choice(svr_C) is executed. I don't think having random in autodoc_mock_imports makes any difference (it is a built-in standard module that is always available). I think the best solution is to put the module-level code in a if __name__ == '__main__' block.

See also Does sphinx run my code on executing 'make html'?.

score 1 · Answer 2 · answered Jul 03 '20 at 01:09

If you didn't use autodoc_mock_imports = ["numpy"] it would have worked without giving an error:

So why did autodoc_mock_imports = ["numpy"] cause the error? Because using mock will cause import numpy as np to provide signatures that although callable are mockups, so their returns will be empty. For example:

import numpy as np
type(np)  # using mock in conf.py
<class 'sphinx.ext.autodoc.mock._MockModule'>

import numpy as np
type(np)  # not using mock in conf.py
<class 'module'>

Quoting the documentation:

autodoc_mock_imports This value contains a list of modules to be mocked up. This is useful when some external dependencies are not met at build time and break the building process. You may only specify the root package of the dependencies themselves and omit the sub-modules:

You don't necessarily have to use mock if your external dependencies (in other words, the imports from outside your own library) work fine at build time when you call Sphinx. Your short example would have worked if you hadn't used mock. Besides using __main__, another common option is not having variables initialize at module level by encapsulating them in a method or function.

6.1. More on Modules A module can contain executable statements as well as function definitions. These statements are intended to initialize the module. They are executed only the first time the module name is encountered in an import statement. (They are also run if the file is executed as a script.)

thanks again! This is true. I removed the if __name__ == '__main__' "numpy" from the mock list and it ran without any errors. This is preferable since it would have been another challenge to integrate __main__ into that part of the code. — D_73, Jul 04 '20 at 05:41

Sphinx autodoc gets stuck on random.choice() using mock

2 Answers2

Linked