-1

I have a bare metal kubernetes cluster set up for my own education purposes. It's three VMs - one master, two workers, running Ubuntu 20.04. I'm having problems with pods forwarding DNS requests.

CoreDNS is running

kubectl get po --all-namespaces |grep dns
kube-system      coredns-74ff55c5b-cdvcv                   1/1     Running   11         4d

It has an expected cluster IP

kubectl get svc kube-dns -n kube-system
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   4d

However it cannot resolve names, such as this attempt on master node:

nslookup github.com 10.96.0.10
Server:     10.96.0.10
Address:    10.96.0.10#53

** server can't find github.com: SERVFAIL

If I edit ConfigMap.yaml

forward . /etc/resolv.conf {
    max_concurrent 1000
}

to forward . 8.8.8.8 { then it will work.

nslookup github.com 10.96.0.10
Server:     10.96.0.10
Address:    10.96.0.10#53

Non-authoritative answer:
Name:   github.com
Address: 140.82.112.3

However this won't resolve private LAN DNS names, obviously. I need CoreDNS to forward to the DNS of the machine it runs on.

I am running k8s 1.20.3, which as I understand no longer has a problem with an infinite loop due to Ubuntu's stub resolv.conf. Besides I've tried brute force deleting the symlink /etc/resolv.conf and recreating it to point to /run/systemd/resolve/resolv.conf on each VM, reboot each, and nothing improved.

I discovered that a pod cannot ping the host machine's DNS (192.168.149.2), so it seems like that's the issue, but I don't know what to do about it.

What else should I be looking at?

jws
  • 109
  • 5
  • Complete guess, in hopes of impacting the iptable, I installed https://kubernetes.io/docs/tasks/administer-cluster/ip-masq-agent/, and this changed something. The pods can now ping other 192.168.149.xxx (worker machine) IPs. But still cannot ping 192.168.149.2 the DNS of these machines. – jws Apr 08 '21 at 17:24
  • 1
    Whats your ClusterIP subnet for pods? – Matt Apr 09 '21 at 07:22
  • The cluster is mostly the defaults of a new install - 10.0.0.0/8 – jws Apr 09 '21 at 11:05
  • 10.0.0.0/8 overlaps 10.96.0.10/16 for serivce subnet. So is it 10.0.0.0/8 or are you just not sure and giving me close but not very specific answer – Matt Apr 09 '21 at 11:14
  • - --service-cluster-ip-range=10.96.0.0/12 – jws Apr 09 '21 at 11:31
  • Also - --cluster-cidr=192.168.0.0/16 in kube-controller-manager.yaml – jws Apr 09 '21 at 11:38
  • Notice your node has 192.168.149.2 which is in the same subnet as your cluster-cidr. It must not be in the same subnet. – Matt Apr 09 '21 at 11:41
  • use different subnet for cluster-cidr. Make sure NONE of subnets in your network overlap. So for example use --cluster-cidr="10.244.0.0/16". To do this you may need to recreate the cluster – Matt Apr 09 '21 at 11:43
  • Do you know if editing /etc/kubernetes/manifests is the best place to make this change? – jws Apr 09 '21 at 11:46
  • It's not that ease. If you want to do it by hand, you may want to have a look at: https://stackoverflow.com/questions/60176343/how-to-make-the-pod-cidr-range-larger-in-kubernetes-cluster-deployed-with-kubead But since you have only 3 vms, its faster to just recreate the cluster – Matt Apr 09 '21 at 11:50

1 Answers1

1

Resolved per Matt's clues.

Pay close attention to initial cluster setup. Instructions online might lead you to overlook an overlapped IP range. Kubernetes private IPs must be separate from private IPs external to the cluster.

Use a tool like kubectl run curl-busybox --image=radial/busyboxplus:curl -i --tty --rm to curl, nslookup and ip a which gives some visibility to the pod's view of the network.

jws
  • 109
  • 5