100 concurrent requests to a Django-App with Gunicorn lead to "54: Connection reset by peer" and 502's

Question

I'm hosting a Django-App which serves as an API-Endpoint. Unfortunately the App which uses the API does a lot of concurrent requests on page-load (in the realm of 80-90 requests).

Nginx is running as the reverse-proxy in front of gunicorn and reports the following problem: kevent() reported that connect() failed (54: Connection reset by peer) while connecting to upstream

That lead me to scaling gunicorn. I tried -k gevent and increasing --worker-connections, but it didn't help much. I had about the same success with --threads and sync-workers.

I noticed a few occurences of Listen queue overflow: 16 already in queue awaiting acceptance (55 occurrences) in dmesg, which lead me to increasing kern.ipc.somaxconn to as much as 4096, but again: Without success.

As per the gunicorn-documentation, I use hey to see if the proxy is doing the right thing and load-testing the Endpoint: hey -c 100 -n 200 -H "Authorization: token ${token}" 'https://example.com/api/'

Which returns something along those lines (sometimes a bit better, sometimes a bit worse):

Status code distribution:
  [200] 126 responses
  [502] 74 responses

Relevant nginx-config:

worker_processes  auto;

events {
    worker_connections  1024;
    accept_mutex on;
    use kqueue;
}

upstream backend-api {
  server 127.0.0.1:8000 fail_timeout=0;
}

server {
    listen 443 ssl http2 accept_filter=httpready;
    server_name example.com;

    ssl_certificate /usr/local/etc/nginx/sites/example.com.crt;
    ssl_certificate_key /usr/local/etc/nginx/sites/example.com.key;

    location = /favicon.ico { access_log off; log_not_found off; }
    root /usr/local/www/example.com/web;
    index index.html ;

    keepalive_timeout 5;

    location ~  ^/(static|assets|index.html|$) {
        root /usr/local/www/example.com/web;
    }

    location / {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Real-IP $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_intercept_errors on;
      proxy_pass http://backend-api;
    }
}

Running gunicorn with: gunicorn --workers=9 --bind 0.0.0.0:8000 --forwarded-allow-ips='*' -k gevent --worker-connections=1000 api.wsgi'

I can't seem to find a solution to this. No matter what I try, I can only get "less" 502's on a page-load, but never reliably none. What am I missing?

Gunicorn does not report any problems - the requests never seem to make it that far.

100 concurrent requests to a Django-App with Gunicorn lead to "54: Connection reset by peer" and 502's

0 Answers0