Nginx gets slow for a specific proxy backend over the time

Question

`I have nginx pod running and making proxy request to different backend.

When it get deployed, it works well for few hours after that it starts taking too much time and getting gateway timeout with 504 status code.

Important observation -

This problem only happens with one of the proxy backend related requests. For proxy request going on other services continues working fine.
If I try out backend service end points directly it continues responding within a fraction of seconds at any point of time.
If I redeploy the nginx pod again, it works perfectly fine again.

Based on above observation, its clear that -

It may not be problem of nginx resource as other proxy backend request are working fine
This is also not the problem backend service getting slow over time as I get response as usual at any point of time.

what could be the issue here?

I have tried to update nginx configuration in different ways. But it doesn't seems working.

Below is snippet of nginx.conf

worker_processes auto;
worker_rlimit_nofile 1024;

events {
accept_mutex on;
accept_mutex_delay 500ms;
worker_connections 1024;
}

http {

include       mime.types;
default_type  application/octet-stream;

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log error;


sendfile on;
server_tokens on;

types_hash_max_size 1024;
types_hash_bucket_size 512;

server_names_hash_bucket_size 256;
server_names_hash_max_size 512;

keepalive_timeout   65s;
keepalive_requests  100;
client_body_timeout 60s;
send_timeout        60s;
lingering_timeout   5s;
tcp_nodelay         on;


client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90s;
proxy_send_timeout      90s;
proxy_read_timeout      90s;
proxy_buffers           32 4k;
proxy_buffer_size       8k;
proxy_set_header        Host $host;
proxy_set_header        X-Real-IP $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header        Proxy "";
proxy_headers_hash_bucket_size 64;

server {
    listen 8080; 

    root /opt/app;

    add_header "X-Frame-Options" "DENY";

    location ~* \.(?:css|js)$ {
        add_header Cache-Control "public, max-age=2592000";
        try_files $uri =404;
    }

    location ~ ^.+\..+$ {
        add_header Cache-Control "public, no-transform, max-age=84600";
        try_files $uri =404;
    }

    location @index {
        add_header Cache-Control "public, max-age=600";
        try_files /index.html =404;
    }
    
    
    location / {
        add_header Cache-Control "public, max-age=600";
        try_files $uri @index;
    }

    
    location ^~ /a {
        proxy_pass            https://serviceahost:443;
    }
    
    location ^~ /b {
        proxy_pass            https://servicebhost:443;
    }           
 }
}

Nginx gets slow for a specific proxy backend over the time

0 Answers0