What else can I do to troubleshoot a package not importing in python yet imports in ipython while in a virtualenv?

Question

I am a Python enthusiast who decided to get serious about creating some of my own reusable packages.

Synopsis of my problem:

1. Calling python test_dummy.py fails to import a package eforest. An ImportError is thrown.
2. import eforest in the python interpreter and in the ipython interpreter both throw no exceptions.
3. Calling run test_dummy.py within the ipython interpreter throws no exception.
4. All cases are run inside of a virtualenv.

virtualenv has nothing to do with my problem. I have posted a new question: script that adds packages to sys.path import as expected when run from ipython but throw exception when the script is run from python

A similar question is at Can't import package from virtualenv

I read the documentation on virtualenv and virtualenvwrapper and installed them both on a Ubuntu 14.04.1 LTS system with Python 2.7.6.

I created a virtualenv:

mkvirtualenv module_troubleshooting -a . -r requirements.txt

My requirements.txt file contained:

Django==1.6.6
EasyProcess==0.1.6
PyVirtualDisplay==0.1.5
South==1.0
argparse==1.2.1
ipython==2.2.0
lxml==3.3.5
selenium==2.42.1
wsgiref==0.1.2

I activated my virtualenv with workon module_troubleshooting

My virtualenv was active: (module_troubleshooting)dmmmd@ubuntuG5:

I understood from the virtualenvwrapper documentation that I could add packages to the sys.path via the add2virtualenv command.

Using add2virtualenv I added my very simple packages I had created. I verified that indeed the pth file contained those packages.

/home/dmmmd/development/python/singleton
/home/dmmmd/development/python/display
/home/dmmmd/development/python/browser
/home/dmmmd/development/python/eforest
/home/dmmmd/development/python/dummy

I ran ipython to see if I could import those packages. I could. No errors. Examining the sys.path in ipython revealed that the above paths were present.

For my own edification I made the following module called test_dummy.py:

import browser
import display
import singleton
import eforest

I ran the following command in ipython: run test_dummy.py.

No exceptions in ipython.

I then exited ipython and ran the following command to assure myself that the global python was NOT being called and to see if python would run the script: /home/dmmmd/.virtualenvs/module_troubleshooting/bin/python test_dummy.py

Traceback (most recent call last):
    File "tests/test_dummy.py", line 4, in <module>
import eforest
ImportError: No module named eforest

Three other packages, browser, display, and singleton imported as expected and the exception was thrown at import eforest

At first, I theorized that eforest might contain some exception. So I ran /home/dmmmd/.virtualenvs/module_troubleshooting/bin/python and had an expected prompt and no import errors when importing eforest:

Python 2.7.6 (default, Mar 22 2014, 22:57:26)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import eforest
>>> print eforest
<module 'eforest' from 'eforest/__init__.pyc'>
>>>

So then I theorized that the sys.path for ipython and python might be different. I saved the sys.path for python to a file and edited it to the following so the list from sys.path could be imported:

/home/dmmmd/.virtualenvs/module_troubleshooting/bin/python -c "import sys; print sys.path" > path.py

I edited path.py so that from path import path would be the list sys.path from calling /home/dmmmd/.virtualenvs/module_troubleshooting/bin/python.

I then ran ipython and typed the following commands:

In [1]: from path import path

In [2]: path # this is the sys.path from calling /home/dmmmd/.virtualenvs/module_troubleshooting/bin/python
Out[2]:
['',
 '/home/dmmmd/development/python/singleton',
 '/home/dmmmd/development/python/eforest',
 '/home/dmmmd/development/python/dummy',
 '/home/dmmmd/development/python/display',
 '/home/dmmmd/development/python/browser',
 '/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7',
 '/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/plat-powerpc-linux-gnu',
 '/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/lib-tk',
 '/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/lib-old',
 '/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-powerpc-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/home/dmmmd/.virtualenvs/module_troubleshooting/local/lib/python2.7/site-packages',
 '/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/site-packages']

To see if ipython's sys.path was different I typed:

In [5]: import sys

In [6]: len(sys.path)
Out[6]: 19

In [7]: len(path)
Out[7]: 16

In [8]: [item for item in sys.path if item not in path]
Out[8]:
['/home/dmmmd/.virtualenvs/module_troubleshooting/bin',
 '/home/dmmmd/.virtualenvs/module_troubleshooting/local/lib/python2.7/site-packages/IPython/extensions',
 '/home/dmmmd/.ipython']

The last two items of ipython's sys.path are ipython related. The one different item is '/home/dmmmd/.virtualenvs/module_troubleshooting/bin'.

Can anybody offer any other troubleshooting advice or explain to me how having '/home/dmmmd/.virtualenvs/module_troubleshooting/bin' in the sys.path in ipython allows eforest to import as expected in ipython when run test_dummy.py is called?

eforest package:

eforest/
├── CustomElementFilters.py
├── CustomEtrees.py
├── CustomEtrees.pyc
├── __init__.py
└── __init__.pyc

Again, eforest imports as expected in ipython and with the python interpreter but it does not import when a script passed as the argument to /home/dmmmd/.virtualenvs/module_troubleshooting/bin/python test_dummy.py.

The only possible exception to eforest that comes to my mind that is different from the other packages is that it imports lxml. Could lxml depend somehow on having '/home/dmmmd/.virtualenvs/module_troubleshooting/bin' in the sys.path?

Any suggestions for troubleshooting would be greatly appreciated. I first noticed the problem when I was using py.test. I thought it had something to do with py.test, but as it turned out the test scripts I was writing didn't run with python either. I decided to stop there and see if I could get some help.

Ruled out virtualenv as a problem source

I ran the following script with python test_dummy.py with the virtuenv deactivated.

import sys
import os
HOME = os.path.expanduser('~')
PYTHON = os.path.join(HOME, 'development/my_python')
PACKAGES = [
    'browser',
    'display',
    'singleton',
    'eforest'
]
for package in PACKAGES:
    package = os.path.join(PYTHON, package)
    if os.path.exists(package) is True:
        if package not in sys.path:
            print "loading package '{0}'".format(package)
            sys.path.append(package)
        else:
            print "package '{0}' already in sys.path".format(package)
    else:
        raise
import browser
import display
import singleton
import eforest

With the virtualenv deactivated eforest is the only package that does not import with running the python test_dummy.py command.

Result:

Traceback (most recent call last):
  File "my_django/automated_browsing/tests/test_dummy.py", line 16, in <module>
import eforest
ImportError: No module named eforest

Answer 1

I found a solution although the solution does not solve the problem of why certain package paths added in an ad hoc fashion to sys.path are not imported when the script is run with python test_dummy.py.

I originally ran into this problem while trying to use py.test. In the py.test documentation I saw this tip about "managing your project with virutalenv, pip, and editable mode." I ignored it because I thought it was too advanced for my level of Python knowledge.

I decided to try the tip after reading about "creating your own python project."

In the pwd I created a setup.py file with the following code:

from setuptools import setup, find_packages

setup(name='import_troubleshooting', version='1.0')
packages = find_packages(exclude=[
    'my_django',
    'fake*',
    'tests*'
])

I then executed the following code at the command line:

pip install -e .  # the pip way (which just calls "setup.py develop")

I then executed the following code with the expected results:

$ python tests/test_dummy.py
# output    
package '/home/dmmmd/development/my_python/browser' already in sys.path
package '/home/dmmmd/development/my_python/display' already in sys.path
package '/home/dmmmd/development/my_python/singleton' already in sys.path
loading package '/home/dmmmd/development/my_python/test_a'
loading package '/home/dmmmd/development/my_python/hello_world'
loading 'browser'
loading 'display'
loading 'singleton'
loading 'test_a'
loading 'hello_world.hello'
Hello, world!

None of this explains to me why I couldn't add the package paths in an ad hoc fashion. I am pleased, however, to have learned something about python installation and packaging. The test_dummy.py script now runs as expected when called from python tests/test_dummy.py.

I am now confused as to why three of the packages are already in sys.path when the test_dummy.py script is run:

package '/home/dmmmd/development/my_python/browser' already in sys.path
package '/home/dmmmd/development/my_python/display' already in sys.path
package '/home/dmmmd/development/my_python/singleton' already in sys.path

I don't recall doing anything that would add those to sys.path but it is possible since I have been doing various tutorials. They are not in sys.path in any other environment where I have not run the setup.py that I created.

My confusion is a symptom of the complexity of Python, not a cause!

Thank you for everybody's input.

NB: This is the output of python test_dummy.py after a fresh virtualenv:

$ pip install -e .
# install output unremarkable
$ python tests/test_dummy.py
# output
loading package '/home/dmmmd/development/my_python/automated_browsing/browser'
loading package '/home/dmmmd/development/my_python/automated_browsing/display'
loading package '/home/dmmmd/development/my_python/automated_browsing/singleton'
loading package '/home/dmmmd/development/my_python/automated_browsing/hello_world'
loading package '/home/dmmmd/development/my_python/automated_browsing/test_a'
loading 'browser'
loading 'display'
loading 'singleton'
loading 'hello_world.hello'
Hello, world!
loading 'test_a'