A curious case of nginx uswgi python

Question

We have a python MVC Web application built using (werkzeug, jinja2 and MongoEngine).

In production we have 4 nginx servers setup behind a nginx load balancer. All 4 servers share a common Mongo server, a Redis server and a Sphinx server.We are using uwsgi between nginx and the application.

Now to the curious case.

Once we deploy a new code, we do a touch xyz.wsgi. For a few hours everything looks fine. but after that we randomly get the error.

 'module' object is not callable

I have seen this error before, in other python development scenarios. But what confuses me this time is the total random behavior.

For Example example.com/multimedia?keywords=sdf&s=title&c=21830.

If we refresh the error is gone. Try another value for any parameter like 'keywords=xzy' and there it is again. Refresh its gone.

That 'multimedia' module is something we did just recently.So we can assume its the root cause. But why does the error occur randomly ?

My assumption is that, it might have something to do with nginx caching or existence of pyc/pyo ? Could a illicit Global Variable be the cause ?

Could you expert hands help me out.

Answer 1

The error probably occurs randomly because it's a runtime error in your code. That is, it doesn't get fired until a user visits your site with the right conditions to follow the code path that results in this error.

It's unlikely to be an nginx caching issue. If it was caching it, then it would probably return the same result over and over rather then change on reload.

However, you can test this by removing nginx and directly testing against werkzeug. Run the requests against it and see if you see the same behavior. No use in debugging Nginx unless you can prove that the underlying systems work the way you expect.

It's also probably worth the 30 seconds to search for module() in your code, since that's the most direct interpretation of that error message.