AppEngine application using Django fails to load

Question

Django is constantly causing our application to crash. After deployment the application is running fine, but once the initial instance is restarted/shutdown it often fails to start with an error similar to the following:


Traceback (most recent call last):  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/wsgi.py", line 236, in call
    self.load_middleware()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/base.py", line 53, in load_middleware
    raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
ImproperlyConfigured: Error importing middleware myfolder.middleware: "No module named myfolder.middleware"

Our file structure is similar to this:


|- app.yaml
|- _ _ init _ _.py
|- settings.py
|- myfolder |
|           |- _ _ init _ _.py
|           |- middleware.py
|           |- ...
|-...
|

Our app.yaml:


application: XXXXX
module: app
version: master
runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /api/(login|logout|passwd|master.|banners.)
  script: app.handler
  secure: always
...

builtins:
- django_wsgi: on

libraries:
- name: django
  version: 1.5

env_variables:
    DJANGO_SETTINGS_MODULE: 'settings'

We have 2 modules in our application and they both exhibit this behaviour (they have similar configurations). Sometimes the modules will stay up for a whole day before crashing again. After they fail to load, all subsequent requests fail with he same error. Deploying one more time always solves the problem temporarily.

We are using plain django with CloudSql. The problem is not reproducible in the development server. After deployment everything in both modules works fine. All middleware, ndb, memcache, cloudsql, taskqueue, etc, including all the modules inside the "myfolder" and every other library xcopied.

The following attempts at solving this problem haven't worked:

We have tried using the appengine_config.py to force django to reload the settings with from django.conf import settings\nsettings._target = None\n
Originally we had shared settings inside "myfolder" and were importing them with "from myfolder.shared_settings import *" inside the root settings.py but django could not load the module myfolder.shared_settings either (similar problem)
using a custom mysettings.py and defining the DJANGO_SETTINGS_MODULE in the app.yaml or in python

The system is not live yet but will be soon and we are running out of options.

Other traces of similarly failing configurations:


Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/wsgi.py", line 236, in __call__
    self.load_middleware()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/base.py", line 45, in load_middleware
    for middleware_path in settings.MIDDLEWARE_CLASSES:
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 48, in _setup
    self._wrapped = Settings(settings_module)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 134, in __init__
    raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'settings' (Is it on sys.path?): No module named myfolder.settings


Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 239, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 353, in __getattr__
    self._update_configs()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 289, in _update_configs
    self._registry.initialize()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 164, in initialize
    import_func(self._modname)
  File "/base/data/home/apps/s~blue-myapp/app:master.375531077560785947/appengine_config.py", line 17, in 
    settings._target = None
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/utils/functional.py", line 227, in __setattr__
    self._setup()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 48, in _setup
    self._wrapped = Settings(settings_module)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 134, in __init__
    raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'settings' (Is it on sys.path?): No module named myfolder.settings

This is our current appengine_config.py:


import sys
import logging

logging.debug(",\n".join(sys.path))

# Since Google App Engine's webapp framework uses Django templates, Django will half-initialize when webapp is loaded.
# This causes the initialization of the rest of Django's setting to be skipped. If you are getting this error, you need
# to explicitly force Django to reload your settings:

from django.conf import settings
settings._target = None

Logging sys.path from appengine_config.py does not change between a successful instance start and a failed instance start (apart from the XXXXXXXXXXX bit of course):


/base/data/home/apps/s~blue-persomi/app:master.3759720XXXXXXXXXXX,
/base/data/home/runtimes/python27/python27_dist/lib/python27.zip,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/plat-linux2,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-tk,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-old,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-dynload,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/site-packages,
/base/data/home/runtimes/python27/python27_lib/versions/1,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/MySQLdb-1.2.4b4,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/protorpc-1.0,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/webob-1.1.1,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/yaml-3.10

Check to see if you are getting a DeadlineExceededError on an instance startup request. If you are not using warmup requests a partially started instance with failed imports can exhibit strange import errors if it gets requests later. — Tim Hoffman, May 20 '14 at 15:48
We are not getting it. It's starting very fast (actually failing very quickly). — miguelv, May 20 '14 at 15:55
What do you mean failing quickly, in your initial post you said instances could run for a day before failing. — Tim Hoffman, May 20 '14 at 15:59
I think you need to elaborate on this statement "Sometimes the modules will stay up for a whole day before crashing again. After they fail to load, all subsequent requests fail with he same error. " What do you mean by "crashing again" ? — Tim Hoffman, May 20 '14 at 16:02
I mean that when the first request to an instance fails it does so in less than 2 secs. All subsequent requests to that instance will continue to fail (60-70ms for these). — miguelv, May 20 '14 at 16:04
Once an instance starts failing, event if I shut it down, the new ones will also fail. — miguelv, May 20 '14 at 16:05
It's the running all day and then crashing again, bit that needs clarifying. I get the subsequant errors occurr — Tim Hoffman, May 20 '14 at 16:05
So what is the "crash"/error that causes the initial failure on the long running instance ? And do you ever have more than one instance running at a time ? I suggest you log sys.path on instance startup, and then trap the ImportError then log sys.path again. — Tim Hoffman, May 20 '14 at 16:06
We don't know what causes it to start failing. Sometimes it's minutes after deployment, other times it's hours. If we shutdown an instance after deployment it starts fine. There is barely no activity in the app since we are only using it for demo purposes so far. The issue with trapping ImportError is that this fails before our code is reached (see first provided stack trace). — miguelv, May 20 '14 at 16:16
Do you have anything in appengine_config.py ? That is run before your code. THere is stuff we are missing here. Is the last request on a working instance the same before the failure ? — Tim Hoffman, May 20 '14 at 16:20
I have update the question with the contents of our appengine_config.py — miguelv, May 20 '14 at 16:24
You can log sys.path in appengine_config.py and that will always be run before any of your requests. That may help you see what is going on. In addition you can wrap a try/except block around app.handler and perform logging there. — Tim Hoffman, May 20 '14 at 16:31
@TimHoffman I have added the sys.path logged on appengine_config.py to the question detail above (both for a successful and failed start). — miguelv, May 20 '14 at 17:07

score 2 · Answer 1 · edited May 23 '17 at 10:24

It seems to be a path related issue as people have mentioned in your question's comments

Possible short-sighted solution add everything to your path manually - look at the top answer here: How to import modules in Google App Engine? At the very least, this will help narrow the problem to path related.

what the docs say: https://developers.google.com/appengine/docs/python/

The Python module include path includes your application's root directory (the directory containing the app.yaml file). Modules you create in your application's root directory are available using a path from the root. Don't forget to create init.py files in sub-directories, so Python will recognize the sub-directories as packages.

so from what I can tell, b/c everything is at or below the app.yaml file in your question the path should already be correct.

double check all your __init__.py file files are in place and spelled correctly.
try deleting all of your *.pyc files and letting them be regenerated.
try importing from the container's folder name FOLDER_CONTAINING_YAML.myfolder.middleware

Thaks a lot. Deleting `*.pyc` files helped. – Sergiy Belozorov Jul 23 '15 at 08:29 — Sergiy Belozorov, Jul 23 '15 at 08:29

score 0 · Answer 2 · answered Oct 17 '14 at 08:35

Add the following lines to your app.yaml

libraries: - name: MySQLdb version: "latest"

It is in the documentation here. https://cloud.google.com/appengine/docs/python/cloud-sql/

Ideally it should have been documented in the GCM DJANGO guide. Would have saved me a lot of time.

Praveen

AppEngine application using Django fails to load

2 Answers2