Nginx server response time is unexpectedly high for one in hundreds of requests

Question

I'm using Nginx to serve .ts files for live streaming (HLS). Everything works fine but there is a problem with the response time of the files. Occasionally, some .ts files received from the server have an unexpectedly high response time which is problematic in our use case.

I have used a python script to find what's the problem.

In this screenshot full response time is the time between the request is sent and the full response is received. The second parameter shown is the time between a request being sent and a response being received. (I've used response.elapsed.total_seconds() in the requests package of python). In slow requests, full response time is higher but response.elapsed.total_seconds() is OK. (And by higher I mean 3-4 times the average response time.)

On top of this problem, I have another problem in which both response time (response.elapsed.total_seconds()) and full response time is high. This case happens less frequently (maybe once in 200 requests) and response times are much higher than in the first case.

I should mention that I have some other machines in front of this machine to proxy and cache files. But doesn't matter if I request the cache machines or the main one. The result is the same.

Here's my Nginx config:

user  root;
worker_processes  16;

worker_rlimit_nofile 1000000;

master_process on;

pid        /run/nginx.pid;

events {
    worker_connections  500000;
    accept_mutex off;
    multi_accept off;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    tcp_nopush on;

    server_tokens off;

    keepalive_timeout  150;
#    keepalive_requests 5;

    aio on;
    gzip on;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 4k;
    client_body_buffer_size 1k;
    client_max_body_size 1k;
    client_body_timeout 12;
    client_header_timeout 12;
    send_timeout 10;
    tcp_nodelay       on;
    sendfile_max_chunk 1m;
    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1.2;
    proxy_intercept_errors on;
    proxy_http_version 1.1;
    vhost_traffic_status_zone shared:vhost_traffic_status:10m;
    underscores_in_headers on;
    recursive_error_pages on;

    server {
        listen 80 backlog=10000000;
        location /tmp {
            access_log /cache/nginx.log main;
            types {
                application/vnd.apple.mpegurl m3u8;
                video/mp2t ts;
            }
            alias /cache/hls;
            add_header Cache-Control no-cache;
     }
}

Nginx access and error logs don't show anything abnormal. Neither for the good responses nor the slow ones.

I couldn't find anything about this problem until now. Hope I could find an answer here.