NGINX is not respecting DNS TTL of Upstream Server

Question

I have an NGINX TCP load balancer with the following configuration:

user myusername;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;
load_module /usr/lib/nginx/modules/ngx_stream_module.so;

events {
    worker_connections 1024;
}

stream {

    upstream api_backend_http {
        server myserver1.mydomain.com:80;
        server myserver2.mydomain.com:80;
    }

    upstream api_backend_https {
        server myserver1.mydomain.com:443;
        server myserver2.mydomain.com:443;
    }

    server {
        listen            80;
        proxy_pass        api_backend_http;
        proxy_buffer_size 16k;
        proxy_connect_timeout 1s;
    }

    server {
        listen            443;
        proxy_pass        api_backend_https;
        proxy_buffer_size 16k;
        proxy_connect_timeout 1s;
    }

    
}

The DNS TTL of myserver1.mydomain.com is set to 30 seconds. 45 Minutes after changing this, NGINX is still sending traffic to the old IP address.

This shouldn't happen - ideally it should respect the TTL of the upstream server DNS name. But it doesn't seem to be doing that. Does anyone know what the actual TTL is, and how to change it?

Side note, this feels like a bug in NGINX.

score 1 · Answer 1 · answered Jul 11 '22 at 06:06

1

You need to explicitly configure nginx to actively and regularly refresh hostname to IP-address mappings.
Otherwise that mapping is only made on restarts /reloads. (That is a shortcoming that happens in many other servers and services too by the way and nothing unique to nginx.)

You need to add a resolver directive and more.

See

answered Jul 11 '22 at 06:06

Rob

1,175
1
7

What if we have multiple upstream servers? I can't get the above example to work if there are multiple upstreams, not just one – A X Jul 13 '22 at 06:07
This doesn't work for multiple upstreams – A X Jul 16 '22 at 23:50

score 0 · Accepted Answer · answered Jul 16 '22 at 23:51

After spending time on this, it seems that you basically have to buy NGINX Plus if you want this and also want to have multiple upstreams.

So consider looking at Envoy. It is free with a great license, and has both DNS refresh and upstream healthchecks built in.

NGINX is not respecting DNS TTL of Upstream Server

2 Answers2