GELF json logs stop arriving on graylog after some time from nginx proxy in docker

Question

Hmm..where to start...i've got an application stack (running in docker swarm), with nginx as proxy in front. Ive got json-formatted logs, being sent to graylog via gelf.

Container system is alpine 3.11.5 and nginx version is 1.17.10

nginx.conf template (we use 1024 worker connetions and 10M body size):

user nginx;
worker_processes auto;

error_log   /var/log/nginx/error.log warn;
pid         /var/run/nginx.pid;

events {
    worker_connections ${MAX_CONNECTIONS};
}

http {
        log_format json escape=json 
            '{'
            '"remoteAddress": "$remote_addr", '
            '"remoteUser": "$remote_user", '
            '"localTime": "$time_iso8601", '
            '"request":"$request", '
            '"requestLength": $request_length, '
            '"requestProcessTime": $request_time, '
            '"responseStatus": $status, '
            '"bodyBytesSent": $body_bytes_sent, '
            '"httpReferer":"$http_referer",'
            '"httpUserAgent": "$http_user_agent",'
            '"http_x_forwarded_for":"$http_x_forwarded_for"'
        '}';

        include /etc/nginx/mime.types;
        default_type    application/octet-stream;

        access_log /var/log/nginx/access.log json;

        sendfile    on;
        #tcp_nopush on;
        server_tokens   off;

        keepalive_timeout   65;

        #gzip   on;

        client_max_body_size ${MAX_BODY_SIZE};

        include /etc/nginx/conf.d/*.conf;
}

conf.d/proxy.conf template

server {
    listen ${Port} default_server;
    listen [::]:${Port} default_server;

    server_name proxy;
    resolver 127.0.0.11 ipv6=off;

    location ~^/help(.*)$ {
        proxy_pass http://help$1$is_args$args;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location / {
        proxy_pass http://webapp/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwared-For $proxy_add_x_forwarded_for;
    }

    location ~ ^/api/(.*?)/(.*)$ {
        proxy_pass http://$1:8080/$1/$2$is_args$args;
        proxy_http_version 1.1;
    }
}

Docker compose logs config:

logging:
        driver: gelf
        options:
          gelf-address: "udp://ip-in-here:12201"
          tag: "dev-proxy"

And it works. Nicely json formatted access logs land on graylog. For a while.

After random ammount of time, access logs are no more, and all that lands on graylog is "an upstream response is buffered to a temporary file while reading upstream". Ive spent good few hours googling stuff, messing around with buffers (disabling proxy_cache, disabling proxy_buffers, increasing proxy_buffers, proxy_buffer_size and proxy_busy_buffers_size), but to no effect. When i disable proxy buffering, there are no log at all. Ofc, stack is working properly and all request are proxied - just no more access log from the proxy.

However, when i change access_log to save to a file (instead of redirecting it to stdout), all access logs save to file, properly formatted.

Updating service / closing proxy container doesnt help. Fully removing stack and deploying it again helps. Until it repeats.

Any ideas what may be the issue here and where should i look?

screenshot

GELF json logs stop arriving on graylog after some time from nginx proxy in docker

0 Answers0