How to make Nginx mirror module not wait for response

Question

We are trying to use Nginx's module ngx_http_mirror_module to mirror traffic to a new webserver of ours.

This seems to lead to latency problem on our prod webserver. After X hours of turning on mirroring, we observe Nginx error logs "2018/07/25 15:55:54 [error] 20#0: *12190535 upstream timed out (110: Connection timed out) while sending to client, client: 10.128.0.37, server: , request: "POST /v1/query?v=20170712 HTTP/1.1", upstream: "http://10.3.248.222:8080//api/query?v=20170712", host: "loadtest.xxx.yyy".

I am speculating that ngx_http_mirror_module may be holding onto both the TCP connection to the original upstream, as well as a connection to the mirroring destination.

Hence, I would like to know how to make Nginx mirror module not wait for response. Or, how to close open sockets to original upstream.

Appreciate any insights!

Answer 1

Your speculations are correct indeed. However, there is a workaround using timeouts. You can explicitly tell Nginx to not wait for the response from the mirrored location after, say 200ms and terminate the connection.

upstream backend {
    server backend.local:10000;
}

upstream test_backend {
    server test.local:20000;
}

server {
    server_name proxy.local;
    listen 8000;

    location / {
        mirror /mirror;
        proxy_pass http://backend;
    }

    location = /mirror {
        internal;
        proxy_pass http://test_backend$request_uri;
        proxy_connect_timeout 200ms;
        proxy_read_timeout 200ms;
    }

}

I'm not sure would be the cost of a terminated connection though, so its better to deploy a lightweight webserver that would queue your request object for later asynchronous consumption.

This does complicate things a bit but its worth noting that you can scale your lightweight web server and your queue and your consumers to match your production demands.