I created a cluster on bare metal, set up with one laptop as the control plane and another laptop as a worker node, for learning purposes. They're in a local lan with a home router.
After a couple of tutorials that went smooth, I started having connectivity / networking issues, and looking on the net brought up more questions than answers.
Now my CoreDNS pods keep crashing: is it expected that they know about the lan router IP (external to the cluster)? Because this is inside one of their logs:
vaio-sve151:~$ kubectl logs -n kube-system coredns-bd6b6df9f-6t25z
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.6
linux/amd64, go1.17.1, 13a9191
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:57243->10.8.191.167:53: i/o timeout
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:58513->10.8.191.167:53: i/o timeout
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:57970->10.8.191.167:53: i/o timeout
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:37901->10.8.191.167:53: i/o timeout
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:55314->10.8.191.167:53: i/o timeout
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:56356->10.8.191.167:53: i/o timeout
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:51310->10.8.191.167:53: i/o timeout
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:39135->10.8.191.167:53: i/o timeout
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:57516->10.8.191.167:53: i/o timeout
[ERROR] plugin/errors: 2 3471238297657131622.6520009371282614310. HINFO: read udp 10.244.1.43:49389->10.8.191.167:53: i/o timeout
[INFO] SIGTERM: Shutting down servers then terminating
[INFO] plugin/health: Going into lameduck mode for 5s
My setup:
@vaio-sve151:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
thinkpad-x260 Ready <none> 2d19h v1.23.17 10.8.191.148 <none> Ubuntu 22.04.2 LTS 5.15.0-73-generic containerd://1.7.2
vaio-sve151 Ready control-plane,master 2d19h v1.23.17 10.8.191.138 <none> Ubuntu 20.04.6 LTS 5.4.0-150-generic containerd://1.7.2
The router (Linksys EA3500) acts as DNS:
@vaio-sve151:~$ nslookup 10.8.191.167
167.191.8.10.in-addr.arpa name = Linksys44105.lan.
Authoritative answers can be found from:
Pods:
vaio-sve151:~$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default bash 1/1 Running 1 (3h29m ago) 3h32m 10.244.1.42 thinkpad-x260 <none> <none>
default kubernetes-bootcamp-65d5b99f84-7zqwb 1/1 Running 0 4h3m 10.244.1.37 thinkpad-x260 <none> <none>
default kubernetes-bootcamp-65d5b99f84-9bm48 1/1 Running 0 4h3m 10.244.1.35 thinkpad-x260 <none> <none>
default kubernetes-bootcamp-65d5b99f84-h69ps 1/1 Running 0 4h3m 10.244.1.36 thinkpad-x260 <none> <none>
kube-flannel kube-flannel-ds-8tv8v 1/1 Running 1 (3h7m ago) 3h17m 10.8.191.138 vaio-sve151 <none> <none>
kube-flannel kube-flannel-ds-tlzz6 1/1 Running 0 3h17m 10.8.191.148 thinkpad-x260 <none> <none>
kube-system coredns-bd6b6df9f-6t25z 0/1 CrashLoopBackOff 49 (4m28s ago) 3h28m 10.244.1.43 thinkpad-x260 <none> <none>
kube-system coredns-bd6b6df9f-8qzs4 0/1 CrashLoopBackOff 49 (65s ago) 3h47m 10.244.1.41 thinkpad-x260 <none> <none>
kube-system etcd-vaio-sve151 1/1 Running 17 (3h7m ago) 2d19h 10.8.191.138 vaio-sve151 <none> <none>
kube-system kube-apiserver-vaio-sve151 1/1 Running 1 (3h7m ago) 4h26m 10.8.191.138 vaio-sve151 <none> <none>
kube-system kube-controller-manager-vaio-sve151 1/1 Running 1 (3h7m ago) 4h25m 10.8.191.138 vaio-sve151 <none> <none>
kube-system kube-proxy-mgzrm 1/1 Running 1 (3h7m ago) 4h17m 10.8.191.138 vaio-sve151 <none> <none>
kube-system kube-proxy-npwrh 1/1 Running 0 4h17m 10.8.191.148 thinkpad-x260 <none> <none>
kube-system kube-scheduler-vaio-sve151 1/1 Running 1 (3h7m ago) 4h24m 10.8.191.138 vaio-sve151 <none> <none>
All the services:
@vaio-sve151:~$ kubectl get services -A -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d19h <none>
default kubernetes-bootcamp NodePort 10.101.18.12 <none> 8080:32682/TCP 3h11m app=kubernetes-bootcamp
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 2d19h k8s-app=kube-dns
Is it normal that a pod like coredns-bd6b6df9f-6t25z has the router IP 10.8.191.167 in its log?
