Where does astroid change its behavior for pylint checks of relative imports?

Question

On a Linux environment, I have the following three files in a folder named utils:

1. An empty file __init__.py

2. A file named mymain.py with the following content:

   from mytool import foo
   

3. A file mytool.py with the following content:

   from subprocess import check_output
   
   def foo():
       print("Hello World")
   

Now I create a virtual environment with the following setup:

astroid==2.4.2
isort==5.7.0
lazy-object-proxy==1.4.3
packaging==20.9
pylint==2.6.0
pyparsing==2.4.7
six==1.15.0
toml==0.10.2
typed-ast==1.4.2
wrapt==1.12.1

Then I run the pylint check on that folder

pylint utils

or

pylint --disable=C,R,W utils  # only the error is of interest

it should return a score of 10.

When you now upgrade the version of astroid to 2.5.0 and run the check again, you get the error:

************* Module utils.mymain
utils/mymain.py:2:0: E0401: Unable to import 'mytool' (import-error)

Is the changed behavior because of the deprecated importlib methods, as pointed out HERE?

Interesting fact

If you remove the unused import statement in the file mytool.py, you also get the same import error even with astroid version 2.4.2

Why; How can an unused import possibly affect the outcome of a pylint check?

Hint: I have two questions.

Answer 1

The problem is not reproducible, so it's hard to say whether the astroid version and the unused import statement are at the heart of the issue, but we can try investigating other avenues in our quest to better understand what's going on here.

---

In a nutshell

Executing pylint --disable=C,R,W utils behaves as if utils is imported as a module, causing the absolute import from mytool import foo to fail.

---

In detail

I believe you expected Pylint to return the perfect score since executing python3 utils/mymain.py from the parent folder or python3 mymain.py from the utils folder work.

Alas, these methods of execution work because sys.path[0] is initialized to the path of the utils folder. Thus, the absolute import statement from mytool import foo succeeds, as the package mytool is found within the paths specified by sys.path.

However, when Pylint is executed, its __main__.py file is treated as the main module and the path of the utils folder is no longer included anywhere within sys.path and the absolute import fails.

Here are three different approaches that cause pylint --disable=C,R,W utils to return the perfect score:

1. Edit the import statement to be relative, i.e. to from .mytool import foo. Note that this change warrants you to execute your script as a module from the parent folder (e.g. python3 -m utils.mymain).

2. Add the utils folder to PYTHONPATH before executing the script or to sys.path before reaching the absolute import statement.

   However, if you do prefer this approach of using an absolute import, I would recommend rearranging the project structure to reflect a well-designed module and importing it properly (e.g. from <module>.utils.mytool import foo). Of course, this requires installing the module locally or manipulating the path to support the import.

3. Try running with Python 2. This works without any need for changes, due to the deprecated behavior of implicit relative imports in Python 2.

---

Addressing your questions on the effect of the astroid module and the unused import statement on the outcome of a Pylint check:

• Since Pylint depends on the astroid module to work, a bug in the astroid module could affect the outcome of a Pylint check in unexpected ways, including in the manner you described.

• Since Pylint was executed on the entire utils folder, the unused import statement in utils/mytool.py could appear as a warning, but it shouldn't alter the outcome of a Pylint check in the manner you described (unless it triggers some unexpected behavior in a buggy astroid version).

Thus, a buggy astroid version, with or without an unused import statement, could yield unexpected results. However, as the problem isn't reproducible, it's hard to research what exactly is going on here.

---

An alternative explanation to this cryptic behavior, suggested by Pylint's official documentation, is an installed module of the same name as the one tested:

You should give Pylint the name of a Python package or module. Pylint will not import this package or module, though uses Python internals to locate them and as such is subject to the same rules and configuration. You should pay attention to your PYTHONPATH, since it is a common error to analyze an installed version of a module instead of the development version.

Answer 2

Try changing import statement

From:

from mytool import foo

To:

from .mytool import foo

The dot in the module name is used for relative module import (Ref Link Here).