Within my AWS environment, I have an nginx server configured to point to an AWS application load balancer's DNS hostname within it's http > upstream backends configuration. All has been working just fine, but recently it would appear that the IP address of the AWS ALB as changed (although it's DNS hostname is immutable) causing my application to fail.
Digging through the nginx log files and checking dig results, it appears that nginx is retaining the IP address of the backend host and not attempting to resolve the IP address every time a request comes in. Once I restart the nginx service, everything starts working again.
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
}
http {
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
client_max_body_size 50M;
upstream backends {
server internal-private-aws-alb-hostname.elb.amazonaws.com:443;
}
server {
listen 443 ssl;
server_name my.servername.com;
ssl_certificate_key /path/to/key.pem;
ssl_certificate /path/to/cert.pem;
ssl_protocols TLSv1.2;
location / {
proxy_pass https://backends;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
}
Has anyone experienced such behavior before and if so, aware of any configuration changes that could be made to make nginx more reliable in this manner? I wondered if there were caching configs that I should be focussing upon, but other than ssl_session_cache shared:SSL:10m; configured in the http section, everything else is rather vanilla.
