Networking not working on Docker / Centos 7.9

Question

I'm running Docker on Centos 7.9.2009 and have a very strange issue. The containers do not have network access and cannot be accessed from the host. I've searched for potential solutions, a lot of which seem to be related to DNS issues (which I don't think is what's going on here since even pinging 8.8.8.8 from within the container does not work). I've tried installing iptables-service, restarting iptables & docker in that order, full reboot etc.

In my efforts to try and find what the problem might be, I ran tcpdump in a separate terminal. As soon as I did, everything works fine! Kill the tcpdump process and it all stops - no network access. Any suggestions of why running tcpdump may resolve the problem? Is it something to do with tcpdump listening on docker0 and establishing a network state?

Output of tcpdump on startup:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes

Output of uname -a:

Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Output of /etc/redhat-release:

CentOS Linux release 7.9.2009 (Core)

Output of ip addr:

3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:8b:94:46:19 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
   valid_lft forever preferred_lft forever
inet6 fe80::42:8bff:fe94:4619/64 scope link
   valid_lft forever preferred_lft forever

Output of iptables --list -t nat:

  Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        anywhere

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Output of docker version:

    Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:03:45 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:02:21 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Thanks in advance!

Answer 1

When tcpdump starts listetning an interface, it puts that interface into promiscuous mode.
That's the immediate reason why your container networking starts working.

Normally docker0 does not require promiscuous mode to work.
Behavior you described indicates problems in your networking stack.

First, your Linux kernel is pretty old.
Version 3.10.0-327 was released in year 2015 by Redhat.
Check for this issue - looks like exactly the same problem.
The author managed to solve it with kernel upgrade and br_netfilter module fix.

Also in couple of other cases (one, two) just removing the interface and Docker restart helped.
But I think you still need to upgrade the kernel - it's too old, really.