Understanding behavior of Python imports and circular dependencies

Question

NOTE: This is about importing modules and not classes, functions from those modules, so I don't think it's a duplicate of the mane "ImportError: cannot import name" results in SO, at least I haven't found one that matches this.

I do understand that importing classes or functions from modules by name might cause a problem, since the module itself might not have been fully initialized yet if there's a circular dependency, but that's not the case here.

In order to reproduce this issue, create three modules that have a circular dependency on it.

First create a package:

$ mkdir pkg
$ touch pkg/__init__.py

Then create pkg/a.py, with contents:

from __future__ import print_function
from __future__ import absolute_import

from . import b

def A(x):
    print('I am A, x={}.'.format(x))
    b.B(x + 1)

def Z(x):
    print('I am Z, x={}. I\'m done now!'.format(x))

And pkg/b.py, with contents:

from __future__ import print_function
from __future__ import absolute_import

from . import c

def B(x):
    print('I am B, x={}.'.format(x))
    c.C(x * 2)

And pkg/c.py, with contents:

from __future__ import print_function
from __future__ import absolute_import

from . import a

def C(x):
    print('I am C, x={}.'.format(x))
    a.Z(x ** 2)

And a main.py (in the top directory) which calls into them:

from __future__ import print_function
from __future__ import absolute_import

from pkg import a

if __name__ == '__main__':
    a.A(5)

I expected there would be no problems with the circular dependency, since there are no references to items within each of the modules during import time (i.e. no references to a.A from modules b or c, except for the call inside the body of c.C).

And, indeed, running this with python3 works just fine:

$ python3 main.py 
I am A, x=5.
I am B, x=6.
I am C, x=12.
I am Z, x=144. I'm done now!

(This is Python 3.5.3 on Debian Stretch, for the record.)

But with python2 (Python 2.7.13), it doesn't really work, and it complains about the circular dependency...

$ python main.py 
Traceback (most recent call last):
  File "main.py", line 5, in <module>
    from pkg import a
  File "/tmp/circular/pkg/a.py", line 5, in <module>
    from . import b
  File "/tmp/circular/pkg/b.py", line 5, in <module>
    from . import c
  File "/tmp/circular/pkg/c.py", line 5, in <module>
    from . import a
ImportError: cannot import name a

So my questions are:

• Why am I running into a circular dependency problem, if I'm not importing or referring to specific classes or functions from my modules, just the modules themselves?

• Why does this only happen on Python 2? (References to PEP, code, release notes or articles about a fix for this in Python 3 would be appreciated.)

• Is there any way to avoid this problem in Python 2, while still not breaking the circular dependency of the modules? I believe not all circular dependencies cause this issue (even in Python 2), so I'm wondering which cases are safe and which cases are not...

Answer 1

When Python starts loading the pkg.a module, it sets sys.modules['pkg.a'] to the corresponding module object, but it only sets the a attribute of the pkg module object at the very end of loading the pkg.a module. This will be relevant later.

---

Relative imports are from imports, and they behave the same. After from . import whatever figures out that . refers to the pkg package, it goes ahead with the regular from pkg import whatever logic.

When c.py hits from . import a, first, it sees that pkg.a is already in sys.modules, indicating that pkg.a has already been loaded or is in the middle of being loaded. (It's in the middle of being loaded, but this code path doesn't care.) It skips to the second part of its job, retrieving pkg.a and assigning it to the a name in the local namespace, but it doesn't just retrieve sys.modules['pkg.a'] to do this.

You know how you can do stuff like from os import open, even though os.open is a function, not a module? That kind of import can't go through sys.modules['os.open'], because os.open isn't a module and isn't in sys.modules. Instead, all from imports, including all relative imports, attempt an attribute lookup on the module they're importing names from. from . import a looks up the a attribute on the pkg module object, but it's not there, because that attribute only gets set when pkg.a finishes loading.

On Python 2, that's it. End of import. ImportError here. On Python 3 (specifically 3.5+), because they wanted to encourage relative imports and this behavior is really inconvenient, from imports try one more step. If the attribute lookup fails, now they try sys.modules. pkg.a is in sys.modules, so the import succeeds. You can see the discussion for this change in the CPython issue tracker at issue 17636.

Answer 2

I am not sure how Python 3 solved the issue but my experience tells Python 2 really can't make it work. The right way to solve the issue is to either:

1. Be careful not to introduce this in your code
2. Do import inside functions, right at the place you need it

Which personally I prefer the latter.

On why, the module system in Python will not mark a module successfully loaded until it is. So at your "import a", Python will not know it already loaded "a" until all dependent loads, "b" and "c" are done as it gone through the whole "a.py" file. So in processing "import c", it will try again to "import a" instead of finding that "a" is something it can skip.