Python import internals difference

Question

I imported a numpy library in two different ways. The first time with from numpy.random import mtrand and the second time after messing with the sys.path.

However, the output of these two module imports was totally different:

>>> from numpy.random import mtrand
>>> dir(mtrand)
['RandomState', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '_rand', 'beta', 'binomial', 'bytes', 'chisquare', 'dirichlet', 'exponential', 'f', 'gamma', 'geometric', 'get_state', 'gumbel', 'hypergeometric', 'laplace', 'logistic', 'lognormal', 'logseries', 'multinomial', 'multivariate_normal', 'negative_binomial', 'noncentral_chisquare', 'noncentral_f', 'normal', 'np', 'pareto', 'permutation', 'poisson', 'power', 'rand', 'randint', 'randn', 'random_integers', 'random_sample', 'rayleigh', 'seed', 'set_state', 'shuffle', 'standard_cauchy', 'standard_exponential', 'standard_gamma', 'standard_normal', 'standard_t', 'triangular', 'uniform', 'vonmises', 'wald', 'weibull', 'zipf']

And the second one:

>>> sys.path.insert(0, '/usr/lib/pymodules/python2.7/numpy/random')
>>> import mtrand
>>> dir(mtrand)
['__builtins__', '__doc__', '__file__', '__name__', '__package__']

How is this behaviour possible?

Edit:

These two tests were executed in different python processes.
Messing with the sys path is stupid, I know that. But this is not for a normal program, it's for an autocompletion. I certainly don't want to import the whole numpy package. I just want to be able to make a dir(mtrand)

This strikes me as the wrong approach to autocompletion. ([Always include your actual aims in the question right from the beginning.](http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem)) You cannot import `mtrand` without also importing NumPy -- for me your second approach also "works", but draws all of NumPy into `sys.modules`, just as a normal import does. — Sven Marnach, Apr 17 '12 at 12:54
Instead of coding this up yourself, you should have a look at [`rlcompleter`](http://docs.python.org//library/rlcompleter.html) from the standard library, [IPython's enhanced version](http://ipython.org/ipython-doc/stable/api/generated/IPython.core.completer.html) of this module and [rope's](http://rope.sourceforge.net/) autocompletion. The latter is the most advanced. — Sven Marnach, Apr 17 '12 at 12:54
rlcompleter and IPython are pretty simple. Rope is the only real autocompletion. But it never worked for me - so I decided to work out something better (all the other solutions are to simple -> python-omnicomplete, or not free -> pycharm). It's already working pretty good, but builtins are still a problem, because they cannot be evaluated. — Dave Halter, Apr 17 '12 at 19:33

score 2 · Answer 1 · answered Apr 16 '12 at 22:50

Trying to import an extension module (i.e. a module that is loaded from a shared library) twice results in undefined behaviour. Quote from the documentation of reload():

In many cases, however, extension modules are not designed to be initialized more than once, and may fail in arbitrary ways when reloaded.

Incidentally, this happens to work fine for my installation of numpy:

>>> from numpy.random import mtrand
>>> dir(mtrand)
['RandomState', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '_rand', 'beta', 'binomial', 'bytes', 'chisquare', 'dirichlet', 'exponential', 'f', 'gamma', 'geometric', 'get_state', 'gumbel', 'hypergeometric', 'laplace', 'logistic', 'lognormal', 'logseries', 'multinomial', 'multivariate_normal', 'negative_binomial', 'noncentral_chisquare', 'noncentral_f', 'normal', 'np', 'pareto', 'permutation', 'poisson', 'power', 'rand', 'randint', 'randn', 'random_integers', 'random_sample', 'rayleigh', 'seed', 'set_state', 'shuffle', 'standard_cauchy', 'standard_exponential', 'standard_gamma', 'standard_normal', 'standard_t', 'triangular', 'uniform', 'vonmises', 'wald', 'weibull', 'zipf']
>>> sys.path.append("/usr/lib/pyshared/python2.7/numpy/random")
>>> import mtrand
>>> dir(mtrand)
['RandomState', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '_rand', 'beta', 'binomial', 'bytes', 'chisquare', 'dirichlet', 'exponential', 'f', 'gamma', 'geometric', 'get_state', 'gumbel', 'hypergeometric', 'laplace', 'logistic', 'lognormal', 'logseries', 'multinomial', 'multivariate_normal', 'negative_binomial', 'noncentral_chisquare', 'noncentral_f', 'normal', 'np', 'pareto', 'permutation', 'poisson', 'power', 'rand', 'randint', 'randn', 'random_integers', 'random_sample', 'rayleigh', 'seed', 'set_state', 'shuffle', 'standard_cauchy', 'standard_exponential', 'standard_gamma', 'standard_normal', 'standard_t', 'triangular', 'uniform', 'vonmises', 'wald', 'weibull', 'zipf']

It's undefined behaviour, so anything can happen.

This might be a good tip, but I imported them in different python instances. -> so that's not the problem. — Dave Halter, Apr 17 '12 at 07:41
Try it in different process and the effect will be the one I described (I guess the above situation is what you described with undefined behaviour) — Dave Halter, Apr 17 '12 at 07:48

score 0 · Answer 2 · answered Apr 16 '12 at 22:50

Not sure why somebody gave you a -1 for this. After having my own problems with Python imports my belief is that when you mess with the sys.path you have bypassed the init.py files that would normally be executed. /usr/lib/pymodules/python2.7/numpy and /usr/lib/pymodules/python2.7/numpy/random are (probably) python packages and contain init.py files. The init files will get included by default whenever anything in that package or child package is imported. By pointing your path to a deeper level of the file tree, you have bypassed these init files, altering the intended behaviour of the package.

Also good info from Sven. I assumed your 2 imports were not within the same running instance of Python. If they were, then Sven's answer is relevant.

Messing around with the sys path is indeed stupid - normally. But since I'm writing an autocompletion, that's really a different thing. I certainly don't want to import the whole `numpy` library. Just the module, to do a `dir(mtrand)` on it (within exec). — Dave Halter, Apr 17 '12 at 07:48

score 0 · Accepted Answer · answered Aug 01 '12 at 13:04

As Endophage pointed out, it is indeed stupid to play with the sys.path. But I guess I started it, because I did not want to execute any Python code, which would be quite cool for an auto-completion. I didn't think that this is a big issue for c_builtin modules. However, there are modules, which really need the correct package path (see the comment about segfaults further down).

I even did a workaround for PyQt4, but noticed that this would not be the only workaround:

sys.path.append('/usr/lib/python2.7/dist-packages/PyQt4')
try:
    import QtCore
except SystemError:
    QtCore = sys.modules['PyQt4.QtCore']
    debug.warning('Loaded a module with SystemError.')

This included catching a SystemError and then using it from sys.modules. Indeed this is very stupid, because I don't know about any side effects of such operations. As I tested PySide (which is another Qt wrapper), segfaults happend under certain circumstances.

So I came up with the more pythonic solution to load modules like this again: from PyQt4 import QtCore and from numpy.random import random.

Interestingly, the memory footprint for loading those modules with their full path, was about as much as messing with the sys.path.

Python import internals difference

3 Answers3