Docker-Compose: Service is not reachable after update

Question

version: '3.7'
services:
  shinyproxy:
    build: /home/administrator/shinyproxy
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname==testnode
    user: root:root
    hostname: shinyproxy
    image: shinyproxy-example
    restart: always
    networks:
      - sp-example-net
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: bind
        source: /home/administrator/shinyproxy/application.yml
        target: /opt/shinyproxy/application.yml
    ports:
      - 4000:4000
  mariadb:
    image: mariadb
    networks:
      - sp-example-net
    volumes:
      - type: bind
        source: /home/administrator/mariadbdata
        target: /var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: root_password
      MYSQL_DATABASE: keycloak
      MYSQL_USER: keycloak
      MYSQL_PASSWORD: xyz
    deploy:
      placement:
        constraints:
          - node.hostname==testnode
  keycloak:
    image: jboss/keycloak
    networks:
      - sp-example-net
    volumes:
      - type: bind
        source: /home/administrator/compose/nginx/fullchain.pem
        target: /etc/x509/https/tls.crt
      - type: bind
        source: /home/administrator/compose/nginx/privkey.pem
        target: /etc/x509/https/tls.key
      - ./theme/:/opt/jboss/keycloak/themes/custom/
    environment:
      - PROXY_ADDRESS_FORWARDING=true
      - KEYCLOAK_USER=xyzasd
      - KEYCLOAK_PASSWORD=xyz
    ports:
      - 8443:8443
    deploy:
      placement:
        constraints:
          - node.hostname==testnode
    restart: "always"
  nginx_service:
    image: nginx_custom
    ports:
      - '80:80'
      - '443:443'
    build: ./nginx/
    networks:
      - sp-example-net


networks:
  sp-example-net:
    driver: overlay
    external: true
    attachable: true

This is my compose file. The keycloak service authenticates shinyproxy users. I use docker-compose up --build -d to get everything running and it workes. Sometimes I have to change small parts of my shinyproxy service and update everything with the same command: Changes get detected and the output looks like this:

compose_keycloak_1 is up-to-date
Recreating compose_shinyproxy_1 ... done

I am running the services in combination with nginx and get the following error:

nginx_service_1  | 2020/06/05 10:02:11 [error] 7#7: *54 connect() failed (113: No route to host) while connecting to upstream, client: 185.130.32.1, server: myserver.com, request: "GET / HTTP/1.1", upstream: "http://10.0.3.181:4000/", host: "myserver.com"

Running docker-compose down and then docker-compose up --build again works, but I do not want to take down all of my services just to update one. Can anyone tell me why this might happen and how to solve it?

Edit: I am more and more sure this is an nginx issue, not a docker issue. Might that be the case?

My nginx.conf looks like this:

server {
  listen                443;
  server_name           server.com;

  ssl on;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
  ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  ssl_certificate /etc/certs/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/certs/privkey.pem; # managed by Certbot


   location / {

    proxy_pass          http://shinyproxy:4000; ### Übernahme der servicenamen aus Docker-compose

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 600s;

    proxy_redirect    off;
    proxy_set_header  Host             $http_host;
    proxy_set_header  X-Real-IP        $remote_addr;
    proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Protocol $scheme;

    }
....

edit2: Issue might be related to this: https://github.com/docker/compose/issues/2003

score 0 · Answer 1 · answered Jun 05 '20 at 10:23

Instead of doing docker-compose down and then docker-compose up --build for the whole project, you can actually, start that particular service by running docker-compose up -d serviceName. Have a look at the example.

-d stands for daemon/detached mode.

version: '3'
services:
  test:
    container_name: test
    image: 'busybox'
    command: 'sleep 5d'
  test1:
    container_name: test1
    image: 'busybox'
    command: 'sleep 4d'

$ docker-compose up -d
Creating network "proj_default" with the default driver
Creating test  ... done
Creating test1 ... done

$ docker-compose ps
Name    Command    State   Ports
--------------------------------
test    sleep 5d   Up           
test1   sleep 5d   Up 

$ docker-compose up -d test1 
Recreating test1 ... done

starting the particular service with the servicename does not work, I still get the error. Docker-compose is actually smart enough to detect the changes and only updates that single service. That is not causing my error unfortunately :-( — Data Mastery, Jun 05 '20 at 11:33

score 0 · Answer 2 · answered Jun 05 '20 at 12:21

The solution was to restart nginx. It seems that like after every restart a container get will a new IP and nginx uses the old IP and will not be able to find it.

restart nginx container when upstream servers is updated

docker exec <nginx_container_id> nginx -s reload

Docker-Compose: Service is not reachable after update

2 Answers2