Docker ports do not seem to be reachable from outside server

Question

I am facing a weird situation with Docker in a SUSE Linux Enterprise Server 12 SP1.

I am connected to the server using SSH. First, I tried to run a simple nginx server to test:

docker run -d -p 8081:80 nginx:alpine --name nginxtest

The container starts successfully. Then, running curl http://localhost:8081 works!

<!-- [...] -->
<title>Welcome to nginx!</title>
<!-- [...] -->

However, when I try to access it from a browser from my computer, at http://10.etc.etc.etc:8081, the request times out.

Surprisingly, though, if I forget about docker for a moment, and use a simple HTTP server directly from the server, such as python3 -m http.server, I get:

Serving HTTP on 0.0.0.0 port 8000 ...

And when I access it from the browser in my computer, at http://10.etc.etc.etc:8000 it works!!

So this must be something about how docker exposes its ports, but it's weird because curl http://localhost:8081 works...

How can I further troubleshoot and then fix this?

Note: Things were working perfectly until yesterday, when I ran systemctl restart wicked (I was trying to investigate another unrelated issue). I also tried restarting the server but it didn't help.

Here is some more output that may be relevant...

ifconfig docker0

docker0   Link encap:Ethernet  HWaddr 02:49:68:4D:40:9B
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:8e6e:ee3f:6918/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:353842611 errors:0 dropped:0 overruns:0 frame:0
          TX packets:450340200 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:509803756314 (486186.7 Mb)  TX bytes:497391388709 (474349.3 Mb)

brctl show docker0

bridge name     bridge id               STP enabled     interfaces
docker0         8000.0249143a607f       no              veth0e4b1f4
                                                        veth1f2fcc2
                                                        veth389f6fe
                                                        [[others]]

docker ps | grep nginx is 0.0.0.0:8081->80/tcp
cat /etc/sysctl.conf

# net.ipv6.conf.all.disable_ipv6 = 1
# net.ipv6.conf.all.disable_ipv6 = 1
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0

(netstat -ltunp | head -2) && (netstat -ltunp | grep docker)

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 :::9000                 :::*                    LISTEN      28216/docker-proxy
tcp        0      0 :::5000                 :::*                    LISTEN      28165/docker-proxy
tcp        0      0 :::8081                 :::*                    LISTEN      30341/docker-proxy

iptables -n --list

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-ISOLATION  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.11          tcp dpt:5000
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.13          tcp dpt:9000
ACCEPT     tcp  --  0.0.0.0/0            172.17.0.2           tcp dpt:80

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

iptables -nt nat -L

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0
MASQUERADE  tcp  --  172.17.0.11          172.17.0.11          tcp dpt:5000
MASQUERADE  tcp  --  172.17.0.13          172.17.0.13          tcp dpt:9000
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:80

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5000 to:172.17.0.11:5000
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:9000 to:172.17.0.13:9000
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8081 to:172.17.0.2:80

docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
20c691cc38e6        bridge              bridge              local
cd95c7d14c38        host                host                local
eb6d8228f366        none                null                local

docker network inspect bridge

[
    {
        "Name": "bridge",
        "Id": "20c691cc38e65be9bf0a377fd8560d49430f523608094b68145e8769e24b1764",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "26887c686d3c8612e460b923692f371bef881065a525496dff9af993ed4b7949": {
                "Name": "sleepy_fermi",
                "EndpointID": "3e79d8c15423b2e145a900e796d316159d2dc51dcc10ced2099b77d1111b03e7",
                "MacAddress": "02:49:68:4D:40:9B",
                "IPv4Address": "172.17.0.6/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

iptables -S

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.11/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5000 -j ACCEPT
-A DOCKER -d 172.17.0.13/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9000 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN

docker version

Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Thu Mar  2 12:26:00 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Thu Mar  2 12:26:00 2017
 OS/Arch:      linux/amd64

Indeed, your service is exposed and reachable through your docker0 interface. Should you want to make it available for external clients, they you'll want to configure something like ip forwarding and some iptables rules (redirection + outbound NAT for docker subnet), sending traffic from your main interface to docker0. Alternatively, set up something like a reverse proxy on your docker host (assuming you only need to expose http/https). If you just need to access it from your browser, then you'll want to connect to some 172.17.x.x address, and not 10.x — SYN, Oct 15 '19 at 14:22
@SYN Thanks for the fast reply! Can you please expand this to an answer? I don't know how to do it. Also, I forgot to say that everything was working until I ran `systemctl restart wicked`... — Pedro A, Oct 15 '19 at 14:26
@SYN note that until yesterday accessing through 10.x was just fine. — Pedro A, Oct 15 '19 at 14:32
Not much familiar with that wicked thing. I'ld assume there's some way to get a persisting configuration out of it... — SYN, Oct 15 '19 at 14:47

score 4 · Answer 1 · answered Oct 15 '19 at 14:46

Here, our Docker container is reachable through docker0 (network: 172.17.x.x)

From your browser, you're trying to connect to some 10.x.x.x address.

Using a Browser on the Docker host

Assuming you just want to connect from your browser to a locally-hosted container, then the easier would just be to reach your container address (172.17.x.x).

In some cases, you could even use ssh -X connecting to your Docker host then start your web browser, without exposing them to your LAN.

Using HTTP Proxy

Assuming you want to expose http/https service to clients on your LAN, then you may use some reverse proxy (nginx, traefik, apache, ...).

dnf install httpd
setsebool -P httpd_can_network_connect on
cat <<EOF >/etc/httpd/conf.d/welcome.conf
<VirtualHost *:80>
    ServerName my-application.example.com
    LogLevel debug
    ErrorLog logs/tunnel_error.log
    CustomLog logs/tunnel_access.log combined
    <Location />
        Require all granted
    </Location>
    ProxyPass / http://<container-address>:8080/
    ProxyPassReverse / http://<container-address>:8080/
</VirtualHost>
EOF
systemctl enable httpd
systemctl start httpd

Configuring Firewall

Assuming you want to expose tcp/udp services to clients on your LAN, you may setup your firewall accordingly:

sysctl -w net.ipv4.conf.all.forwarding=1
echo net.ipv4.conf.all.forwarding=1 >>/etc/sysctl.conf
iptables -A FORWARD -i docker0 -o eth0 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -t nat -A PREROUTING -p tcp -d <my-eth0-address> --dport 8080 -j DNAT --to <my-container-address>:8080

Note those iptables rules would not persist, if you reboot your system, or somehow reset your firewall. Not much familiar with Wicked.

Hello, thank you very much for the help so far! I would like to try the Firewall option, but I have a question, the last command you suggested above mentions the port 8080, is this specific port special? Or am I supposed to replace it with the ports in my containers? So if I have several containers I should run several of those commands? Can't I just "open everything"? — Pedro A, Oct 15 '19 at 16:11
Hello, I added the outputs of more commands in my question, such as `netstat -ltunp`, `iptables --list`, etc. Hopefully they are helpful. My current thoughts are that I shouldn't need a per-port configuration, since it worked previously just fine... — Pedro A, Oct 15 '19 at 17:39

Docker ports do not seem to be reachable from outside server

1 Answers1

Using a Browser on the Docker host

Using HTTP Proxy

Configuring Firewall