DNS Resolution Failure on Kubernetes/Calico with MySQL setup

Question

Ive hit a wall I'm hoping the SO community can advise on where to go next. I've set up a 6 node kubernetes cluster with calico as the networking service. I've only got two pods, the dns debugging pods from kubernetes and a mysql pod. Well and the kube-system pods. Anyways, I've been at this all day. I've started from scratch 3 times and I keep hitting a wall when it comes to DNS. I've been trying to sort through why I can't access my pods externally. Here are my configs.

mysql.yaml

kind: Service
metadata:
  name: mysql
  namespace: new_namespace
spec:
  type: ExternalName
  externalName: mysql.new_namespace.svc.cluster.local
  ports:
  - port: 3306
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: new_namespace
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: *******
        securityContext:
         runAsUser: 0
         allowPrivilegeEscalation: false
        ports:
        - name: mysql
          containerPort: 3306
          protocol: TCP
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: gluster-claim

Along with others, I've been primarily following this guide, but have been unsuccessful in determining my issue. DNS issues just... make no sense to me.

$ kubectl exec -i -t -nnew_namespace dnsutils -- nslookup mysql
Server:     192.168.128.10
Address:    192.168.128.10#53

*** Can't find mysql.new_namespace.svc.cluster.local: No answer

It seems like things should be working...

$ kubectl exec -i -t -nnew_namespace dnsutils -- nslookup kubernetes.default
Server:     192.168.128.10
Address:    192.168.128.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 192.168.128.1

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS      AGE
kube-system   calico-kube-controllers-58497c65d5-mcmb4   1/1     Running   0             31m
kube-system   calico-node-27ln4                          1/1     Running   0             36m
kube-system   calico-node-dngfs                          1/1     Running   6 (39m ago)   45m
kube-system   calico-node-nq6bz                          1/1     Running   0             38m
kube-system   calico-node-p6jwz                          1/1     Running   0             35m
kube-system   calico-node-p8fzr                          1/1     Running   0             35m
kube-system   calico-node-wlzr9                          1/1     Running   0             35m
kube-system   calico-typha-68857595fc-kgnvx              1/1     Running   0             45m
kube-system   calico-typha-68857595fc-n4hhq              1/1     Running   0             45m
kube-system   calico-typha-68857595fc-vjgkc              1/1     Running   0             45m
kube-system   coredns-78fcd69978-25bxh                   1/1     Running   0             26m
kube-system   coredns-78fcd69978-cfl52                   1/1     Running   0             26m
kube-system   etcd-new_namespace-master                         1/1     Running   3             49m
kube-system   kube-apiserver-new_namespace-master               1/1     Running   0             49m
kube-system   kube-controller-manager-new_namespace-master      1/1     Running   0             31m
kube-system   kube-proxy-4zx4m                           1/1     Running   0             35m
kube-system   kube-proxy-hhvh7                           1/1     Running   0             38m
kube-system   kube-proxy-m8sph                           1/1     Running   0             35m
kube-system   kube-proxy-qrfx7                           1/1     Running   0             49m
kube-system   kube-proxy-tkb4m                           1/1     Running   0             35m
kube-system   kube-proxy-vct78                           1/1     Running   0             36m
kube-system   kube-scheduler-new_namespace-master               1/1     Running   3             49m
new_namespace        dnsutils                                   1/1     Running   0             30m
new_namespace        mysql-554fd8859d-hb7lp                     1/1     Running   0             4m5s

$ kubectl get svc --all-namespaces
NAMESPACE     NAME              TYPE           CLUSTER-IP        EXTERNAL-IP                      PORT(S)                  AGE
default       kubernetes        ClusterIP      192.168.128.1     <none>                           443/TCP                  51m
kube-system   calico-typha      ClusterIP      192.168.239.47    <none>                           5473/TCP                 47m
kube-system   kube-dns          ClusterIP      192.168.128.10    <none>                           53/UDP,53/TCP,9153/TCP   51m
new_namespace        gluster-cluster   ClusterIP      192.168.180.197   <none>                           1/TCP                    30m
new_namespace        mysql             ExternalName   <none>            mysql.new_namespace.svc.cluster.local   <none>                   31m

$ kubectl get endpoints --all-namespaces
NAMESPACE     NAME              ENDPOINTS                                                      AGE
default       kubernetes        10.1.0.125:6443                                                52m
kube-system   calico-typha      10.1.0.126:5473,10.1.0.127:5473,10.1.0.128:5473                48m
kube-system   kube-dns          192.168.13.1:53,192.168.97.65:53,192.168.13.1:53 + 3 more...   52m
new_namespace        gluster-cluster   10.1.0.125:1,10.1.0.126:1,10.1.0.127:1 + 3 more...             31m

$ kubectl describe endpoints kube-dns --namespace=kube-system
Name:         kube-dns
Namespace:    kube-system
Labels:       k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=CoreDNS
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-09-06T15:30:06Z
Subsets:
  Addresses:          192.168.13.1,192.168.97.65
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns-tcp  53    TCP
    dns      53    UDP
    metrics  9153  TCP

Events:  <none>

And the logs...don't really mean anything to me. It looks like things are working though? Yet I still can't access mysql..

$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns                                                                                                                                                                                      new_namespace-master: Mon Sep  6 16:01:47 2021

[INFO] 192.168.119.1:52410 - 18128 "A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 97 0.00009327s
[INFO] 192.168.119.1:41837 - 46102 "A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000240183s
[INFO] 192.168.119.1:42485 - 36923 "A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000097762s
[INFO] 192.168.119.1:54354 - 34171 "A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000155781s
[INFO] 192.168.119.1:36491 - 48004 "A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 141 0.000075232s
[INFO] 192.168.119.1:58078 - 26522 "A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000096242s
[INFO] 192.168.119.1:59389 - 51728 "AAAA IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 141 0.000110561s
[INFO] 192.168.119.1:39553 - 24302 "A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000114412s
[INFO] 192.168.119.1:60340 - 28351 "A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 141 0.000175322s
[INFO] 192.168.119.1:36494 - 12725 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 106 0.000149332s
[INFO] 192.168.119.1:45875 - 25210 "AAAA IN mysql. udp 23 false 512" NXDOMAIN qr,rd,ra,ad 98 0.000629398s
[INFO] 192.168.119.1:37467 - 44662 "A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 97 0.000115082s
[INFO] 192.168.119.1:37792 - 59085 "AAAA IN mysql. udp 23 false 512" NXDOMAIN qr,aa,rd,ra 98 0.000043841s
[INFO] 192.168.119.1:47263 - 56267 "AAAA IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 141 0.000241662s
[INFO] 192.168.119.1:47070 - 59276 "A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 141 0.000144522s
[INFO] 192.168.119.1:46812 - 32557 "A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 141 0.00008474s
[INFO] 192.168.119.1:57113 - 14895 "AAAA IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 141 0.000052781s
[INFO] 192.168.119.1:51403 - 18192 "AAAA IN mysql.new_namespace.svc.cluster.local. udp 48 false 512" NOERROR qr,aa,rd 141 0.000166912s
[INFO] 192.168.119.1:52077 - 43229 "A IN kubernetes.default.new_namespace.svc.cluster.local. udp 61 false 512" NXDOMAIN qr,aa,rd 154 0.000199803s
[INFO] 192.168.119.1:60907 - 16052 "AAAA IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 147 0.000082662s

Oh and this was the init command I used:

sudo kubeadm reset ; sudo kubeadm init --pod-network-cidr=192.168.0.0/17 --service-cidr=192.168.128.0/17 --apiserver-advertise-address 10.1.0.125  --control-plane-endpoint 10.1.0.125

Looks like your externalName: mysql.default.svc.cluster.local.lan field is wrong as the externalName is referancing to a CNAME record being an alias of an existing domain and it seems like mysql.default.svc.cluster.local.lan isn't resolvable by an external DNS server. You can read more about externalName here: https://kubernetes.io/docs/concepts/services-networking/service/#externalname. — Jakub Siemaszko, Sep 06 '21 at 15:57
Thanks Jakub - I removed the .lan and tried a different namespace. See the updated post :) — The Kaese, Sep 06 '21 at 16:09

score 1 · Answer 1 · answered Sep 07 '21 at 02:28

Turns out I just hadn't installed Calico correctly. I started from scratch one more time and installed calico using this guide (NOT THE QUICKSTART), and everything's working. I'm not really sure how I came across finding out calico was installed incorrectly. I think it was because I had two calico controllers in two different namespaces and I was very confused as to why. Anyways, reran the same init command, followed the guide, set up mysql and I have access!

DNS Resolution Failure on Kubernetes/Calico with MySQL setup

1 Answers1