3

I have setup kubernetes in ubuntu 16.04. I am using kube version 1.13.1 and using weave for networking. I have initialized the cluster using :

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.88.142

and weave:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

All the pods seems to be running fine but coredns always remains in CrashLoopBackOff status. I have read mostly all the solution available for this. 

NAME                                READY   STATUS             RESTARTS   AGE
coredns-86c58d9df4-h5plc            0/1     CrashLoopBackOff   7          18m
coredns-86c58d9df4-l77rw            0/1     CrashLoopBackOff   7          18m
etcd-tx-g1-209                      1/1     Running            0          17m
kube-apiserver-tx-g1-209            1/1     Running            0          17m
kube-controller-manager-tx-g1-209   1/1     Running            0          17m
kube-proxy-2jdpp                    1/1     Running            0          18m
kube-scheduler-tx-g1-209            1/1     Running            0          17m
weave-net-npgnc                     2/2     Running            0          13m

I initially started by editing the cordens file and deleting the loop. It resolves the issue but then later I realized that I wasn't able to ping www.google.com from within the container but I was able to ping the ip address of google.com. Thus deleting the loop is not a perfect solution.

Next I tried looking at the /etc/resolv.conf and found below contents:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search APSDC.local

Here is the workaround provided on kubernetes page which says that any type IP address like 127.0.0.1 should be avoided. I am not able to understand this line as this file is automatically generated. How can make changes to the file so that coredns can work fine. Belo is the logs of coredns:

$ kubectl logs coredns-86c58d9df4-h5plc -n kube-system

.:53
2019-01-31T17:26:43.665Z [INFO] CoreDNS-1.2.6
2019-01-31T17:26:43.666Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
[INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
[FATAL] plugin/loop: Forwarding loop detected in "." zone. Exiting. See https://coredns.io/plugins/loop#troubleshooting. Probe query: "HINFO 1423429973721138313.4523734933111484351.".

Can anyone please point me to right direction in order to resolve this issue. Please help. Thanks

S Andrew
  • 5,592
  • 27
  • 115
  • 237

2 Answers2

3

I have resolved this issue. In my case I had below contents of /etc/resolv.conf

nameserver     127.0.1.1

I first used the below command to get the correct IP as the device was in client's network.

nmcli device show <interfacename> | grep IP4.DNS

After this I updated the file /etc/resolvconf/resolv.conf.d/head with below contents

nameserver    192.168.66.21

and then run the below command to regenerate the resolv.conf

sudo resolvconf -u

After this I had below contents in /etc/resolv.conf:

nameserver    192.168.66.21
nameserver    127.0.1.1

I then deleted the coredns pods and everything worked fine. Thanks.

S Andrew
  • 5,592
  • 27
  • 115
  • 237
  • This solution should be the best but it didn't work for me. I changed the `configmap/coredns`, to point directly to my upstream resolver. See this https://blog.heptio.com/configuring-your-linux-host-to-resolve-a-local-kubernetes-clusters-service-urls-a8c7bdb212a7 – debiasej Mar 14 '19 at 18:43
  • How did you update a resolvconf file in a failed pod? I ended up editing configmap replacing /etc/resolv.conf with 8.8.8.8 (anyway it's a sandbox so far). – Drey Sep 03 '21 at 14:02
1

I had the same problem in ubuntu 16.04. My /etc/resolv.conf pointed to loopback address too. I first tried as S Andrew to change the resolv.conf file manually but although without any luck.

In my case, I have resolved this issue disabling dnsmasq. In /etc/NetworkManager/NetworkManager.conf, comment on the following line in the [main] section:

[main]
#dns=dnsmasq

Dnsmasq makes it simple to specify the nameserver to use for a given domain but sets automatically a loopback address and this makes coredns to crash.

After this the /etc/resolv.conf file pointed to my DNS Servers supplied by my ISP.

debiasej
  • 980
  • 1
  • 14
  • 26