How to troubleshoot - ImportError: Could not import settings 'mysite.settings' when deploying django?

Question

I've had my django app deployed and working perfectly with apache and mod_python, according to the apache deployment instructions. But since I changed the project structure somewhat, I haven't been able to get it working. No matter what I try, I keep getting the following page:

MOD_PYTHON ERROR

ProcessId:      27841
Interpreter:    '127.0.0.1'

ServerName:     '127.0.0.1'
DocumentRoot:   '/var/www'

URI:            '/myapp/'
Location:       '/myapp/'
Directory:      None
Filename:       '/var/www/myapp'
PathInfo:       '/'

Phase:          'PythonHandler'
Handler:        'django.core.handlers.modpython'

Traceback (most recent call last):

  File "/usr/lib/python2.6/dist-packages/mod_python/importer.py", line 1537, in HandlerDispatch
    default=default_handler, arg=req, silent=hlist.silent)

  File "/usr/lib/python2.6/dist-packages/mod_python/importer.py", line 1229, in _process_target
    result = _execute_target(config, req, object, arg)

  File "/usr/lib/python2.6/dist-packages/mod_python/importer.py", line 1128, in _execute_target
    result = object(arg)

  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/core/handlers/modpython.py", line 212, in handler
    return ModPythonHandler()(req)

  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/core/handlers/modpython.py", line 174, in __call__
    self.load_middleware()

  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/core/handlers/base.py", line 39, in load_middleware
    for middleware_path in settings.MIDDLEWARE_CLASSES:

  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/utils/functional.py", line 276, in __getattr__
    self._setup()

  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)

  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/conf/__init__.py", line 89, in __init__
    raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))

ImportError: Could not import settings 'settings' (Is it on sys.path?): No module named settings

Reading various threads, the problem could be one of the following

• incorrect apache configuration file
• incorrect sys.path
• incorrect permissions on any file or directory
• bugs in my django site code

Is there any way to narrow the problem down some? For example, is there a way to find out what sys.path is at the time of error? Are there any log files that can help me here? Is there any way to distinguish between "file not found" and "no permission to read file"? I'd like to approach the solution more scientifically than just trying random things until I stumble upon the solution.

This question is not the same as: Mod_python error: ImportError: Could not import settings. I'm asking specifically for ways to help narrow down the possible solutions, rather than the solutions themselves.

Answer 1

Ok, I figured out the problem myself, so I'll answer my own question. I find that there is no apache log or any other source of information to figure out what exactly is causing the Import Error. So I share my method below, as a more "rational" approach, in the hope that I can save somebody a few frustrating hours of randomly changing stuff.

For the record, here is the apache configuration that I used.

<Location "/mysite/">
    SetHandler python-program
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE mysite_django.settings
    PythonOption django.root /mysite
    PythonDebug On
    PythonPath "['/home/mysite/prg/mysite/mysite_django', '/home/mysite/prg/mysite'] + sys.path"
</Location>

I created a user "mysite" on the server, and the django settings.py is located in the (somewhat redundantly named) location /home/mysite/prg/mysite/mysite_django/settings.py. I open an ssh channel to the server. In this case I use the very useful screen utility to switch quickly between users (but a second ssh channel would work just as well)

screen

Press Ctrl-A, Ctrl-C to create a new screen. Now it's important to see the permissions exactly as apache does, so I switch to the user that runs apache, which is www-data on debian systems:

sudo -i -u www-data

I'm going to try to import settings.py myself from the python environment:

python

>>> import os, sys
>>> print sys.path
... result omitted ...

This obviously should give you the standard sys.path without any of the "mysite" directories, because we're running plain python. So I modified sys.path as follows. I copy-pasted directly from the apache configuration, to avoid typos:

>>> sys.path = ['/home/mysite/prg/mysite/mysite_django', '/home/mysite/prg/mysite'] + sys.path
>>> import mysite_django.settings
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named mysite_django.settings

As expected, I get the same error I already saw when loading my site in a browser. So I can replicate the problem within the python environment, that is a good thing. Clearly, the build path should be formed from the sys path entry '/home/mysite/prg/mysite' plus the requested import path 'mysite_django/settings.py'. So why is it not importing? Let's make sure the given sys path entry actually exists

>>> os.path.exists (sys.path[0])
False

It doesn't exist? Strange.

>>> os.path.exists ('/home')
True
>>> os.path.exists ('/home/mysite')
True
>>> os.path.exists ('/home/mysite/prg/')
True
>>> os.path.exists ('/home/mysite/prg/mysite')
True
>>> os.path.exists ('/home/mysite/prg/mysite/mysite_django')
False

Here I realize that there is something wrong with the permissions of mysite_django. Probably a left-over from all the random changes that I made. But actually, the problem is in the directory above, mysite, because that is where it tries to read the directory listing from. I switch to the superuser screen with Ctrl-A, Ctrl-N, and I use chmod to set the permissions of mysite so that they are the same as the last directory that succeeded:

$ ls -la /home/mysite/prg
total 12
drwxr-xr-x 3 mysite mysite 4096 2011-11-08 11:24 .
drwxr-xr-x 7 mysite mysite 4096 2011-11-08 14:56 ..
drwxr-xr-- 6 mysite mysite 4096 2011-11-08 11:35 mysite
$ chmod o+x /home/mysite/prg/mysite/
$ ls -la /home/mysite/prg
total 12
drwxr-xr-x 3 mysite mysite 4096 2011-11-08 11:24 .
drwxr-xr-x 7 mysite mysite 4096 2011-11-08 14:56 ..
drwxr-xr-x 6 mysite mysite 4096 2011-11-08 11:35 mysite

Switching back to the www-data screen with Ctrl-A, Ctrl-N:

>>> os.path.exists ('/home/mysite/prg/mysite/mysite_django')
True
>>> os.path.exists ('/home/mysite/prg/mysite/mysite_django/settings.py')
True

Now for the final test

>>> import mysite_django.settings
>>>

Yay! Success We can even go further and test that django can load

>>> from django.core.management import setup_environ
>>> setup_environ(mysite_django.settings)
'/home/mysite/prg/mysite/mysite_django'

At this point the site was working again in my browser.

By following these steps, you can at least ensure that:

• there are no permission problems for settings.py for user www-data
• there are no mistakes in the PythonPath setting of your apache configuration
• there are no runtime errors when importing settings.py

Answer 2

for my Django 1.5 and mod_wsgi, I resolved it like this:

httpd.conf:

WSGIScriptAlias /mysite "/var/www/html/mysite/mysite/wsgi.py"
WSGIPythonPath  "/var/www/html/mysite/mysite"
<Directory "/var/www/html/mysite/mysite">
    Order deny,allow
    Allow from all
</Directory>

wsgi.py:
the sys.path.append(path) to let mysite.settings known

import os
import sys                                              #new
path = '/var/www/html/mysite'                           #new
if path not in sys.path:                                #new
    sys.path.append(path)                               #new

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

another thing, to avoid error like below

Premature end of script headers: wsgi.py

you need remove cgi configure from httpd.conf

AddHandler cgi-script .py