How to fast failover nginx upstream containers in docker (blue/green deployment)

Question

How can I get dockerized nginx to failover quickly when one of the upstream containers goes down? Failover seems instantaneous in a non-docker environment, but several requests timeout when dockerized.

I'm using nginx as a load balancer/proxy on a single node/vm in front of two instances of the same node app. I pictured new version deployments to go as follows (often called blue-green deployments):

1. New version of the node app pushed to github
2. Docker hub builds a new image
3. One of the upstream containers is taken down, upgraded, and redeployed
4. As one of the containers is taken down, nginx automatically fails over to the other live container
5. Once the upgraded container is deployed, the other container is taken offline for upgrade/redeployment as well

However, on nginx doesn't seamlessly do #4. When I take down one of the upstream containers for version bump, nginx will timeout several requests (in round-robin fashion) as it figures out one of the containers is down. I haven't had this happen in a non-docker environment.

Here's my docker cloud stack file:

load-balancer:
  image: 'foo/load-balancer:latest'
  links:
    - node-blue
    - node-green
  ports:
    - '80:80'
node-blue:
  image: 'foo/node-app:latest'
node-green:
  image: 'foo/node-app:latest'

nginx.conf looks like:

events {
  worker_connections  1024;
}

http {
  gzip on;

  upstream app {
    server node-green;
    server node-blue;
  }

  server {
    listen 80;
    server_name app.local;
    location / {
        proxy_pass http://app;
    }
  }
}

Answer 1

After a bunch of wrestling, I think I came out with a reasonable solution. Swapped out a custom nginx container for dockercloud/haproxy. Apparently the load balancer needs to be able to listen to docker cloud events, and adjust accordingly.

Stack file now looks like:

load-balancer:
  image: 'dockercloud/haproxy:1.5.3'
  restart: always
  roles:
    - global
  links:
    - node-blue
  ports:
    - '80:80'
node-blue:
  image: 'foo/node-app:latest'
  environment:
    - VIRTUAL_HOST=app.local
node-green:
  image: 'foo/node-app:latest'
  environment:
    - VIRTUAL_HOST=app.local

And when I want to deploy a new version, I just run the following script:

docker-cloud service set --link node-green:node-green load-balancer
docker-cloud service redeploy node-blue --sync
docker-cloud service set --link node-blue:node-blue load-balancer
docker-cloud service redeploy node-green --sync