NGINX requires restart when service reset

Question

We use NGINX in docker swarm, as a reverse proxy. NGINX sits within the overlay network and relays external requests on to the relevant swarm service.

However we have an issue, where every time we restart / update or otherwise take down a swarm service, NGINX returns 502 Bad Gateway. NGINX then continues to serve a 502 even after the service is restarted, and this is not corrected until we restart the NGINX service, which obviously defies the whole point of having a load balancer and services running in multiple places.

Here is our NGINX CONF:

events {}
http {
  fastcgi_buffers 16 16k;
  fastcgi_buffer_size 32k;
  client_max_body_size 20M;

  large_client_header_buffers 8 256k;
  client_header_buffer_size 256k;

  proxy_buffer_size          128k;
  proxy_buffers              4 256k;
  proxy_busy_buffers_size    256k;

  map $host $client {
     default clientname;
  }

  #Healthcheck
  server {
    listen 443;
    listen 444;
    location /is-healthy {
      access_log off;
      return 200;
    }
  }

 #Example service:
  server {
     listen 443;
     server_name scheduler.clientname.com;

     location / {
        resolver 127.0.0.11 ipv6=off;
        proxy_pass http://$client-scheduler:60911;
        proxy_http_version 1.1;

        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
     }
  }

  #catchll 
  server {
     listen 443;
     listen 444;
     server_name _;

     location / {     
         return 404 'Page not found';
     }
  }
}

We use the $client placeholder as otherwise we can't even start nginx when one of the services is down.

The other alternative is to use an upstream directive that has health checks, which can work well. Issue with this is that if any of the services are unavailable, NGINX won't even start!

What are we doing wrong?

UPDATE It appears what we want here is impossible (please prove me wrong though!). Seems crazy to miss such a feature in the world of docker and micro-services!

We are currently looking at HAPROXY as an alternative, as this can be setup with default-server init-addr none to stop failure on startup.

Same issue: Nginx can't connect to gateway after a microservice is restarted. Both Microservice and Nginx are dockerized. — Vahid F, Nov 30 '19 at 08:41
Yes we never found the solution to this. We now have a process to restart nginx every time a microservice restarts, which kind of defies the whole point. We are in the process of migrating to HA proxy for this reason. — Paul Grimshaw, Dec 02 '19 at 08:14

score 0 · Answer 1 · answered Dec 09 '19 at 05:55

Here is how I do it, create an upstream with max_fails=0

    upstream docker-api {
        server docker.api:80 max_fails=0;
    }


    # load configs
    server {
        listen 80;
        listen [::]:80;

        server_name localhost;

        location /api {
            proxy_pass http://docker-api;
            proxy_http_version  1.1;
            proxy_cache_bypass  $http_upgrade;

            # Others config...
        }

    }

score 0 · Answer 2 · edited Mar 24 '23 at 17:28

I had the same problem by using docker-compose. Nginx container could not connect the web service after docker-compose restart.

Finally I figure out two circumstances cause this glitch. First, docker-compose restart do not follow the depends_on which should be restart the nginx after web restarted. Second, docker-compose restart reassign a new internal IP address to containers and nginx do not refresh the web IP address after it start up.

My solution is define a variable to force nginx resolve the IP every time:

location /api {
    set $web_service "http://web_container_name:13579";
    proxy_pass $web_service;
}

NGINX requires restart when service reset

2 Answers2