So I've got this Django application (more specifically, Django Rest Framework) up there on Google's app engine. Mostly it works fine, except once in a while, I hit a strange delay (from 4-5s to 10-20 seconds...) This delay happens while authenticating the user and before I hit the view itself. I have already crossed out two theories :
- Delays caused by starting and setting up a new instance : I'm already using the warmup option for python instances (https://cloud.google.com/appengine/docs/python/config/appconfig#Python_Warmup_requests)
- Cloud SQL-related delays. The application talks with Cloud SQL to authenticate the user, although I've logged every SQL queries and the delay happens before any queries are made.
So if it's not related to GAE's architecture and not limited by RPC... what's happening? I've plugged in cProfile and here's what it can tells us (I wrapped cProfile around the wsgi handler) :
Please note the line in the middle : around 10000 "len" calls. This same profiling report is devoid of such a line, usually. Only these requests with this absurd delay contains these 10000s of "len" calls.
This might indicate a very long query to Cloud SQL, but as I've said beforehand, the queries logged happens after. Or there is maybe a glitch in the SQL logging... Maybe.