Getting Connection Refused from Docker Swarm on Ubuntu 20.04

Question

Update below!

I am puzzled by Docker swarm on Ubuntu 20.04. I created a clean Ubuntu machine on Scaleway and basically followed the tutorial from https://dockerswarm.rocks/ . I soon found this tutorial also, which is a little shorter and cleaner: https://blog.creekorful.com/2019/10/how-to-install-traefik-2-docker-swarm/ .

Basically (all from SSH'ing into the machine):

• Installed Docker
• Initialized a Swarm with # docker swarm init --advertise-addr=x.x.x.x, where I used the remote IP address of the server.
• I then created an overlay network # docker network create --driver=overlay my-net
• I started a simple hello world container:

version: '3'
services:
  helloworld:
    image: tutum/hello-world:latest
    ports:
            - 80:80
    networks:
     - my-net
networks:
    my-net:
       external: true

# docker stack deploy -c helloworld.yml helloworld

• From this point I'd assume I should be able to # curl 127.0.0.1 and get my hello world. However, I get a connection refused: curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused

Things I tried/checked

Output from docker service ls

root@www2:~# docker service ls
ID                  NAME                    MODE                REPLICAS            IMAGE                      PORTS
md1bd2ydswo8        helloworld_helloworld   replicated          1/1                 tutum/hello-world:latest   *:80->80/tcp

Output from docker ps

root@www2:~# docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED              STATUS              PORTS               NAMES
7c2d9d7379c5        tutum/hello-world:latest   "/bin/sh -c 'php-fpm…"   About a minute ago   Up About a minute   80/tcp              helloworld_helloworld.1.7u99ox2ea6bylb5by8vdca0pt

Enabling UFW on ubuntu

UFW was disabled by default, but I also tried enabling it. This is the output of # ufw status

root@www2:~# ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere                  
2376/tcp                   ALLOW       Anywhere                  
2377/tcp                   ALLOW       Anywhere                  
7946/tcp                   ALLOW       Anywhere                  
7946/udp                   ALLOW       Anywhere                  
4789/udp                   ALLOW       Anywhere                  
80/tcp                     ALLOW       Anywhere                  
22/tcp (v6)                ALLOW       Anywhere (v6)             
2376/tcp (v6)              ALLOW       Anywhere (v6)             
2377/tcp (v6)              ALLOW       Anywhere (v6)             
7946/tcp (v6)              ALLOW       Anywhere (v6)             
7946/udp (v6)              ALLOW       Anywhere (v6)             
4789/udp (v6)              ALLOW       Anywhere (v6)             
80/tcp (v6)                ALLOW       Anywhere (v6)

Messing around with the IP addresses

I also tried curling to the public IP and the private IP, as well as using the private IP as advertise-addr. I also tried curling from my laptop to the remote server, both by IP and domain name. But all to no avail...

Trying on Arch Linux

I run Arch Linux on my laptop and tried to do the same locally on my laptop, which has no problems whatsoever.

Any ideas are very welcome, thanks!

Update

So as I pretty much lost it, I decided to spin up another VPS, clean Ubuntu 18.04 this time (as opposed to 20.04) and it worked right away, no problems whatsoever........

Answer 1

I had been banging my head against this issue for a few days. On a fresh Ubuntu 20.04 image on IBM Cloud, unable to expose a port from a Docker Swarm service.

I eventually found this message in the logs from sudo journalctl -u docker.service

level=error msg="Could not get ipvs family information from the kernel. It is possible that ipvs is not enabled in your kernel. Native loadbalancing will not work until this is fixed."

This was confirmed by installing ipvsadm with sudo apt-get install ipvsadm, then running sudo ipvasdm, which returns a similar message.

I switched back to 18.04 rather than faff about trying to rebuild a kernel, but no doubt that is possible should you be so inclined.

Answer 2

Just my 2 cents. I had a similar behaviour, but this was due to the container not properly starting up, because of an issue with the underlying filesystem for my volume bind mounts.

To ensure you're not in that case, just start your stacks with local volumes only.

Also, make sure the packets are well reaching your swarm nodes. You can check with combinations of iptables commands such as:

iptables -Z #to Zero all counters
iptables -v -L #to list rules & packet counts
iptables -v -t nat -L #to also check in the NAT table. The NAT table is probably the most relevant here, since I guess the first rules applied are in the prerouting section in NAT table.

Regards,