I have a Docker swarm running Docker version 1.13.1. I am regularly deploying stacks of Docker services (via docker stack deploy) to this swarm, and I have one nginx proxy service that sits at ports 80 and 443 acting as a reverse proxy to various applications in the swarm.
I ran into a problem with using nginx's upstream capability was that it cached the DNS lookup of my service names. This worked fine for a while but as more stacks were removed and deployed those cached IP addresses became stale and nginx would start timing out or serving requests to the wrong container.
I attempted to fix this using the following technique:
[in nginx.conf]
server {
server_name myapp.domain.com;
resolver 127.0.0.11 valid=10s ipv6=off;
set $myapp http://stack_myapp:80; # stack_myapp is the DNS name of the service
location / {
proxy_pass $myapp;
}
}
# other similar server blocks
127.0.0.11 appears to be the IP address of the internal DNS server the swarm sets up. This seems to work most of the time - the IP addresses of the upstream services do not get cached for long and the proxy recovers if upstream services move around. However, the proxy will occasionally still serve requests to incorrect addresses, for example, it will serve requests to http://10.0.0.12:80/... and time out or hit the wrong container. When I run docker exec proxycontainer ping stack_myapp, I get the correct IP address. Why is nginx not resolving the correct IP when ping does?
