2
$ nmap localhost

Starting Nmap 6.40 ( http://nmap.org ) at 2019-02-12 12:59 +00
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0027s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 995 closed ports
PORT    STATE SERVICE
22/tcp  open  ssh
25/tcp  open  smtp
80/tcp  open  http
111/tcp open  rpcbind
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.23 seconds
$ sudo netstat -lnt 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN     
tcp6       0      0 :::111                  :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 ::1:25                  :::*                    LISTEN     
$

Why are 80 and 443 not captured by netstat?

ss does not report the missing ports either. This is found on a centos 7 box. Both 80 and 443 are actually open and working as nmap found out -- curl from another host can pull stuff as expected.

The special thing is that 80 and 443 are opened by a docker container running on this host (the commands were run on the host, not in the container, just to be clear). The other 3 (22, 25, 111) are by non-docker local programs. I'm guessing docker is doing some voodoo but I have been unable to locate anything useful.

Vin
  • 559
  • 2
  • 18
  • what is the result of `docker ps` ? – Mostafa Hussein Mar 17 '19 at 17:01
  • I can't get the output for you right now. But if you are looking for open ports, `docker ps` reported correct open ports on the container. – Vin Mar 17 '19 at 17:03
  • 1
    80 and 443 should be appearing in the netstat output if you have mapping to the container and then expected to appear in nmap's output – Mostafa Hussein Mar 17 '19 at 17:09
  • are you doing port maping for 80 and 443? with `docker run -p 80:80 -p443:443` ? – Mostafa Hussein Mar 17 '19 at 17:11
  • Yes, it was `-p 80:80 -p 443:443`. – Vin Mar 17 '19 at 17:14
  • So maybe something related with the connection status then ? – Mostafa Hussein Mar 17 '19 at 17:21
  • Behavior-wise, they are listening -- as I said, I'm able to perform HTTP requests from another host just fine, and `nmap` can find them as well. I tried `netstat -a` as well, with no luck. If you mean other "connection status", do help to elaborate how I can check on them. – Vin Mar 17 '19 at 17:26
  • no that what i meant indeed, try with `netstat -an` again and add it to your question – Mostafa Hussein Mar 17 '19 at 20:21
  • The output is very noisy. Are you looking for anything specific in `netstat -a`? I'm unable to grep any related `:80` or `:443`. – Vin Mar 21 '19 at 15:11
  • I tried many times to reproduce the issue but i couldn't, ports should always appear. maybe you have some kind of customization ? – Mostafa Hussein Mar 21 '19 at 15:22
  • 1
    We have migrated to k8s, and `NodePort` for exposing ports. And now it's registering ports correctly in the `netstat`. For now I can't get the prior deployment live to try reproducing it :( But once I can, or I find a way to do it with experiments, I'll update with more details. I'm still very curious about what's going on. Thank you so much @MostafaHussein for all the time and help you shared! – Vin Mar 27 '19 at 15:41
  • To Alex R: not reproducible on a fresh Centos7 VM. See the details: https://i.imgur.com/ogi3tc2.png – anemyte Apr 21 '22 at 14:54
  • @anemyte I'm using an Alpine image with docker running on Amazon Linux 2, but everything else is the same – Alex R Apr 23 '22 at 04:40

1 Answers1

2

As of v1.7 docker has a configuration flag --userland-proxy which can be set to true or false (I believe this can be set to false be default these days). Basically what it does (being set to false) is instead of using a proxy process to get the ingressing traffic to the container, it utilizes the iptables rules to nat/forward the traffic (hairpin NAT) to the container.

See this article for more detailed explanation. From what I was able to gather, in most cases when the userland-proxy is disabled the port will still show up in the netstat, but this is only to allocate the port, so that the other host applications wouldn't be able to bind it, but the actual data plane follows the rules specified in the iptables. At the same time I've came across a bug when it wasn't the case and the port didn't show up in the output of the netstat/ss.

I believe this is what is happening in your case. You do not see the port in the output of the netstat, but the traffic still can get to the container because userland-proxy is disabled and iptables magic is used to get there.

jabbson
  • 4,390
  • 1
  • 13
  • 23
  • I caught this in my logs... it seems consistent with the "iptables magic" you're talking about - `[INFO] Running command /bin/sh -c iptables -A OUTPUT -t nat -o lo -p tcp --dport 80 -j DNAT --to 172.17.0.2:80 -m comment --comment added_by_elasticbeanstalk` – Alex R Apr 23 '22 at 04:53