Django, low requests per second with gunicorn 4 workers

Question

I'm trying to see why my django website (gunicorn 4 workers) is slow under heavy load, I did some profiling http://djangosnippets.org/snippets/186/ without any clear answer so I started some load tests from scratch using ab -n 1000 -c 100 http://localhost:8888/

A simple Httpreponse("hello world") no middleware ==> 3600req/s

A simple Httpreponse("hello world") with middlewares (cached session, cached authentication) ==> 2300req/s

A simple render_to_response that only print a form (cached template) ==> 1200req/s (response time was divided by 2)

A simple render_to_response with 50 memcache queries ==> 157req/s

Memcache queries should be much faster than that (I'm using PyLibMCCache)? Is the template rendering as slow as this result?

I tried different profiling technics without any success.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 46936
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 400000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 46936
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ sysctl -p

fs.file-max = 700000
net.core.somaxconn = 5000
net.ipv4.tcp_keepalive_intvl = 30

I'm using ubuntu 12.04 (6Go of ram, core i5)

Any help please?

It depends ... what do you do in the views? what queries do you run ? what do you run them on ? — ionelmc, Mar 07 '13 at 16:40
is your memcached server on the same machine or on an external one ? — Tommaso Barbugli, Mar 09 '13 at 13:06

score 1 · Answer 1 · answered Mar 09 '13 at 13:40

It really depends on how long it takes to do a memcached request and to open a new connection (django closes the connection as the request finishes), both your worker and memcached are able to handle much more stress but of course if it takes 5/10ms to do a memcached call then 50 of them are going to be the bottleneck as you have the network latency multiplied by call count.

Right now you are just benchmarking django, gunicorn, your machine and your network.

Unless you have something extremely wrong at this level this tests are not going to point you to very interesting discoveries.

What is slowing doing your app is very likely to be related to the way you use your db and memcached (and maybe at template rendering).

For this reason I really suggest you to get django debug toolbar and to see whats happening in your real pages.

If it turns out that opening a connection to memcached is the bottleneck you can try to use a connection pool and keep the connection open.

I did all that stuff (memcache-debug-panel, profiling...), basically iIsolved my problem by minimizing the memcache queries and storing the html code directly on memcache (because some django template tags are very slow also). by doing this I multiplied the r/s * 100 — surfeurX, Mar 09 '13 at 18:56

score 0 · Answer 2 · answered Jan 29 '14 at 13:46

You could investigate memcached performance.

$ python manage.py shell
>>> from django.core.cache import cache
>>> cache.set("unique_key_name_12345", "some value with a size representative of the real world memcached usage", timeout=3600)
>>> from datetime import datetime
>>> def how_long(n):
        start = datetime.utcnow()
        for _ in xrange(n):
            cache.get("unique_key_name_12345")
        return (datetime.utcnow() - start).total_seconds()

With this kind of round-trip test I am seeing that 1 memcached lookup will take about 0.2 ms on my server.

The problem with django.core.cache and pylibmc is that the functions are blocking. Potentially you could get 50 times that number in the round trip for HTTP request. 50 times 0.2 ms is already 10 ms.

If you were achieving 1200 req/s on 4 workers without memcached, the average HTTP round-trip time was 1/(1200/4) = 3.33 ms. Add 10 ms to that and it becomes 13.33 ms. The throughput with 4 workers would drop to 300 req/s (whuch happens to be in the ballpark of your 157 number).

Django, low requests per second with gunicorn 4 workers

2 Answers2