script that adds packages to sys.path import as expected when run from ipython but throw exception when the script is run from python

Question

I am a perplexed Python enthusiast. I posted What else can I do to troubleshoot a package not importing in python yet imports in ipython while in a virtualenv? thinking that I was having a problem with my virtualenv. Some more troubleshooting has revealed this is probably not the case.

I would appreciate any insight or troubleshooting tips so that I can get on with package development. Thank you in advance!

Synopsis of my problem on Ubuntu 14.04.1 LTS system with Python 2.7.6.:

1. I have a script named test_dummy.py that adds package paths to sys.path and then attempts to import the packages.
2. The packages import without error when the script is run with run test_dummy.py inside of ipython.
3. ImportError exceptions are thrown for certain packages when the script is run with python test_dummy.py.
4. What is most frustrating is that the packages that contain code i.e. more than just an __init__.py file import without exception. Simple test packages with only a __init__.py file in them do not import as expected. I cannot determine what is causing the simplest of packages to throw exceptions on import when the script is called from python i.e. python test_dummy.py.

My pwd is as such:

.
├── browser
├── display
├── hello_world
├── singleton
├── test_a
└── tests

test_a is as such:

test_a
├── __init__.py
└── __init__.pyc

hello_world is as such:

hello_world/
├── hello.py
├── hello.pyc
├── __init__.py
└── __init__.pyc

hello.py is as such:

HELLO = 'Hello, world!'
print HELLO

The other packages ['browser', 'display', 'singleton'] all contain init.py files and other code that throws no known exceptions when run.

tests is as such:

tests
└── test_dummy.py

The code in test_dummy.py:

import importlib
import os
import sys
HOME = os.path.expanduser('~')
PYTHON = os.path.join(HOME, 'development/my_python')
PACKAGES = [
    'browser',
    'display',
    'singleton',
    'test_a',
    'hello_world',
]
MODULES = [
    '',
    '',
    '',
    '',
    'hello',
]
IMPORTS = ['.'.join((pkg, module)).strip('.')
           for pkg, module
           in zip(PACKAGES, MODULES)]
"""
append sys.path with PACKAGES if package exists and not already
in the sys.path
"""
for package in PACKAGES:
    package = os.path.join(PYTHON, package)
    if os.path.exists(package) is True:
        if package not in sys.path:
            print "loading package '{0}'".format(package)
            sys.path.append(package)
        else:
            print "package '{0}' already in sys.path".format(package)
    else:
        message = "Package '{0}' does not exist.".format(package)
        raise IOError(message)
"""
import IMPORTS if the package is in sys.path
"""
for item in IMPORTS:
    pkg_in_path = [pkg == os.path.basename(path)
                   for pkg in PACKAGES for path in sys.path]
    if any(pkg_in_path):
        print "loading '{0}'".format(item)
        importlib.import_module(item)
    else:
        raise AttributeError("{0} not in sys.path".format(item))

ipython results:

$ ipython
# result
Python 2.7.6 (default, Mar 22 2014, 22:57:26)
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
Warning: disable autoreload in ipython_config.py to improve performance.

In [3]: run tests/test_dummy.py
loading package '/home/dmmmd/development/my_python/browser'
loading package '/home/dmmmd/development/my_python/display'
loading package '/home/dmmmd/development/my_python/singleton'
loading package '/home/dmmmd/development/my_python/test_a'
loading package '/home/dmmmd/development/my_python/hello_world'
loading 'browser'
loading 'display'
loading 'singleton'
loading 'test_a'
loading 'hello_world.hello'
Hello, world!

results from python tests/test_dummy.py:

loading package '/home/dmmmd/development/my_python/browser'
loading package '/home/dmmmd/development/my_python/display'
loading package '/home/dmmmd/development/my_python/singleton'
loading package '/home/dmmmd/development/my_python/test_a'
loading package '/home/dmmmd/development/my_python/hello_world'
loading 'browser'
loading 'display'
loading 'singleton'
loading 'test_a'
Traceback (most recent call last):
  File "tests/test_dummy.py", line 46, in <module>
    importlib.import_module(item)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named test_a

Importing the package 'test_a' in the python interpreter does not throw an exception:

Python 2.7.6 (default, Mar 22 2014, 22:57:26)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/home/dmmmd/development/my_python/test_a')
>>> import test_a
>>> print test_a
<module 'test_a' from 'test_a/__init__.pyc'>
>>>

Answer 1

I found a solution although the solution does not solve the problem of why certain package paths added in an ad hoc fashion to sys.path are not imported when the script is run with python test_dummy.py.

I originally ran into this problem while trying to use py.test. In the py.test documentation I saw this tip about "managing your project with virutalenv, pip, and editable mode." I ignored it because I thought it was too advanced for my level of Python knowledge.

I decided to try the tip after reading about "creating your own python project."

In the pwd I created a setup.py file with the following code:

from setuptools import setup, find_packages

setup(name='import_troubleshooting', version='1.0')
packages = find_packages(exclude=[
    'my_django',
    'fake*',
    'tests*'
])

I then executed the following code at the command line:

pip install -e .  # the pip way (which just calls "setup.py develop")

I then executed the following code with the expected results:

$ python tests/test_dummy.py
# output    
package '/home/dmmmd/development/my_python/browser' already in sys.path
package '/home/dmmmd/development/my_python/display' already in sys.path
package '/home/dmmmd/development/my_python/singleton' already in sys.path
loading package '/home/dmmmd/development/my_python/test_a'
loading package '/home/dmmmd/development/my_python/hello_world'
loading 'browser'
loading 'display'
loading 'singleton'
loading 'test_a'
loading 'hello_world.hello'
Hello, world!

None of this explains to me why I couldn't add the package paths in an ad hoc fashion without first installing my packages with pip install -e .. It is strange, too, that the failure is only when passing the script to python. ipython test_dummy.py works without installing my local packages first.

I am pleased, however, to have learned something about python installation and packaging. The test_dummy.py script now runs as expected when called from python tests/test_dummy.py.

I am now confused as to why three of the packages are already in sys.path when the test_dummy.py script is run:

package '/home/dmmmd/development/my_python/browser' already in sys.path
package '/home/dmmmd/development/my_python/display' already in sys.path
package '/home/dmmmd/development/my_python/singleton' already in sys.path

I don't recall doing anything that would add those to sys.path but it is possible since I have been doing various tutorials. They are not in sys.path in any other environment where I have not run the setup.py that I created.

My confusion is a symptom of the complexity of Python, not a cause!

Thank you for everybody's input.

NB: This is the output of python test_dummy.py after a fresh virtualenv:

$ pip install -e .
# install output unremarkable
$ python tests/test_dummy.py
# output
    loading package '/home/dmmmd/development/my_python/automated_browsing/browser'
loading package '/home/dmmmd/development/my_python/automated_browsing/display'
loading package '/home/dmmmd/development/my_python/automated_browsing/singleton'
loading package '/home/dmmmd/development/my_python/automated_browsing/hello_world'
loading package '/home/dmmmd/development/my_python/automated_browsing/test_a'
loading 'browser'
loading 'display'
loading 'singleton'
loading 'hello_world.hello'
Hello, world!
loading 'test_a'