Is it possible to make subpackage appear as the actual package

Question

I came around this question and got quite disappointed by how the tensorflow developers try to make the tensorflow directory appear as the actual package, whereas the actual package root is actually tensorflow/python. By using a __init__.py file of the form

from tensorflow.python import *
del python

they try to achieve this goal. This results in some inconsistent behaviour (at least so it seems to me) when working with the package, e.g.

import tensorflow.python                # seems to work
tensorflow.python                       # AttributeError: no attribute 'python'
from tensorflow.python import Session   # works as expected
tensorflow.python.Session               # AttributeError: no attribute 'python'
from tensorflow import python           # works as expected

tensorflow.nn                           # works as expected
import tensorflow.nn                    # ImportError: no module 'tensorflow.nn'
tensorflow.nn.tanh                      # works as expected
from tensorflow.nn import tanh          # ImportError: no module 'tensorflow.nn'

Now, I was wondering whether/how it could be possible to avoid most/all of these issues to get a more consistent behaviour. The first set of inconsistencies could be easily resolved by not deleting the python attribute. However, given that the goal would be to make the complete package appear as if it is a sub-package, this might not be entirely satisfactory.

To keep things simple, let's consider the following package structure

package/
    __init__.py
    api/
        __init__.py
        code.py

where package/api/code.py looks something like

def a(): 
    return 'alpha'
def b(): 
    return 'bravo'

and package/api/__init__.py would be

import package.api.code

would it be possible to create package/__init__.py so that the following works

import package.api                      # ImportError: no module 'package.api'
package.api                             # AttributeError: no attribute 'api'
from package.api import code            # ImportError: no module 'package.api'
package.api.code                        # AttributeError: no attribute 'api'
from package import api                 # ImportError: cannot import 'api'

package.code                            # works as expected
import package.code                     # works as above
package.code.a                          # works as expected
from package import a                   # correctly imports function a

I believe that the last four lines of code should give the expected result by adding to sys.modules, but I do not seem to be able to find a way to make import package.api fail.

Would anyone have an idea on how this could be done? Feel free to point me to use-cases that I am overlooking or should consider to achieve the above-mentioned goal.

Answer 1

First of all - I must say that I literally hate "shady" techniques like this. It has a bad effect on various IDE intellisense and makes the library structure less understandable. But still...

If you want the submodule code.py to act as an actual subpackage, you need to create a dummy module:

package/
    __init__.py
    api/
        __init__.py
        code.py
    code/
        __init__.py

Add this in code/__init__py:

from package.api.code import *

And this in package/__init__.py:

from package.code import *

And then this part should work as intended:

import package.code            # works as expected
package.code                   # works as expected
package.code.a                 # works as expected
from package import a          # works as expected

If you further add this to the package/__init__.py:

import package.api
del package.api

You basically disconnect user from accessing package.api, but nothing else, and they can still access the submodule through subpackage using 'from x import y':

import package.api            # works
package.api.a()               # AttributeError: module 'package' has no attribute 'api'

import package.api.code       # works
package.api.code.a()          # AttributeError: module 'package' has no attribute 'api'

from package.api import code  # works
code.a()                      # works

from package import api       # works
api.code.a()                  # AttributeError: module 'package.api' has no attribute 'code'

Answer 2

I managed to write something that almost works (in package/__init__.py):

import sys

from package.api import *

for key in sys.modules:
    parts = key.split('.')
    if len(parts) > 1 and parts.pop(0) == __name__:
        subkey = parts.pop(0)
        if subkey == 'api' and len(parts) == 0:
            sys.modules['.'.join([__name__, subkey])] = None
        elif subkey == 'api':
            m = sys.modules.pop(key)
            sys.modules['.'.join([__name__] + parts)] = m
del api        
del sys

The import errors suggest that it is still quite a hack, but apart from that most all of the examples work as specified iff the package has already been loaded once (i.e. if import package or alike has been invoked before running the statements from my question). If the first statement is import package.api, there is thus no ImportError as I would like.

In an attempt to find a solution for this problem, I stumbled upon this answer, which practically leads to the same behaviour with much more elegant code:

import sys

from package import api

# clean up this module
self = sys.modules.pop(__name__)
del self

# this module becomes hidden module
sys.modules[__name__] = api
sys.modules[api.__name__] = None

del api
del sys

However, this still suffers from the problem that if the first import is something like import package.api, no ImportError is thrown.