How docker container reaches to embedded dns server

Question

I'd like to ask one question regarding docker's network mechanism.

context/what I've got to understand

※ I use one example container and one network(bridge) made by these commands

docker network create mynw
docker run -itd --name app --network mynw ubuntu tail -f /dev/null

1. In container, when we query to dns servers, packets of those requests are forwarded into 127.0.0.11:{random port per container}

• containers queries to 127.0.0.11:53 by default

root@c23eae78f2c7:/app# cat /etc/resolv.conf
nameserver 127.0.0.11
options edns0 trust-ad ndots:0

• those requests are forwarded to 127.0.0.11:{random port per container} by iptable's rule

root@c23eae78f2c7:/app# iptables-legacy -nvL DOCKER_OUTPUT -t nat
Chain DOCKER_OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:36369
   13  1019 DNAT       udp  --  *      *       0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:53199

2. In container, I can see there is an active connection listining on the port

(Let's say, the random port is 53199 now)

• there is an active connection listing on the port

root@c23eae78f2c7:/app# ss -au
State           Recv-Q          Send-Q                   Local Address:Port                      Peer Address:Port          Process
UNCONN          0               0                           127.0.0.11:53199                          0.0.0.0:*

• Actually, I can query to the port directly

root@c23eae78f2c7:/app# dig @127.0.0.11 -p 53199 db

; <<>> DiG 9.16.22-Debian <<>> @127.0.0.11 -p 53199 db
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24918
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;db.                IN  A

;; ANSWER SECTION:
db.         600 IN  A   172.18.0.2

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53199(127.0.0.11)
;; WHEN: Wed Dec 15 05:05:03 UTC 2021
;; MSG SIZE  rcvd: 38

3. In host, dockerd has logs that says it's resolving hostnames/container names

• we can see these lops. So I believe embedded dns server is working as a part of dockerd process

root@srv2:/proc/588662# journalctl -p debug --follow -u docker

# (in container) $ dig app
Dec 15 05:06:37 ubuntu dockerd[583979]: time="2021-12-15T05:06:37.763719372Z" level=debug msg="Name To resolve: app."
Dec 15 05:06:37 ubuntu dockerd[583979]: time="2021-12-15T05:06:37.763784750Z" level=debug msg="[resolver] lookup for app.: IP [172.18.0.3]"

# (in container) $ dig stackoverflow.com

Dec 15 05:07:15 ubuntu dockerd[583979]: time="2021-12-15T05:07:15.715601264Z" level=debug msg="Name To resolve: stackoverflow.com."
Dec 15 05:07:15 ubuntu dockerd[583979]: time="2021-12-15T05:07:15.715708196Z" level=debug msg="[resolver] query stackoverflow.com. (A) from 127.0.0.1:46570, forwarding to udp:127.0.0.53"

4. In host, I can see an opened file descriptor

• I can find some open file descriptiros type of which is sock, and name is protocol: UDP or protocol: TCP
  • As many as containers I create, where are pairs of udp/tcp file descriptors

root@srv2:/home/ubuntu/docker/ch5# ps aux | grep dockerd
root      583979  0.1  0.7 1532280 90208 ?       Ssl  02:29   0:12 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

root@srv2:/home/ubuntu/docker/ch5# lsof -p 583979 | grep -E "(UDP|TCP)"
dockerd 583979 root   31u     sock                0,9      0t0    3165293 protocol: UDP
dockerd 583979 root   33u     sock                0,9      0t0    3165294 protocol: TCP

• I also believe these file descriptors are corresponding to active connections found in containers

What's the problem

• However, I cannot find How dockerd is working in host, but listening in containers.

What I'd like to ask

• If the context so far is correct,
  • I'd like to get the last piece: "How dockerd is working in host, but listening in containers"
• If the context is not correct, I'd appreciate if I could get some hints

ref: other Q&A I've read

• how does Docker Embedded DNS resolver work?
  • I'm not sure what this sentence means: The Sandbox for each container allows to route DNS queries through the network namespaces.
  • I'd like to know how docker achieve Binding the resolver to the container's internal interface is the reason why you haven't found any process on the host here

I'd be very happy if I can get any help or hint on this topic. Thank you in advance!

Answer 1

I also notice these questions when I troubleshoot docker DNS server questions!

Docker will used iptables redirect dns query to dockerd daemon.

-A OUTPUT -d 127.0.0.11/32 -j DOCKER_OUTPUT
-A POSTROUTING -d 127.0.0.11/32 -j DOCKER_POSTROUTING
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 127.0.0.11:45981
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:38095
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p tcp -m tcp --sport 45981 -j SNAT --to-source :53
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 38095 -j SNAT --to-source :53

When an endpoint joins the network, dockerd will setup a dns Server!

Just create a socket and listen in container netns. This listen socket accepts in host netns, handle by dockerd!

https://xie.infoq.cn/article/6bb2727db4ebff3f1e0ced82d