kubernetes external-dns not updating dns: "HTTP request failed: Get [url] : x509: certificate is valid for [router domain], not api.cloudflare.com"

Question

I'm trying to set up external-dns as outlined in the docs for a bare-metal microk8s server, using Cloudflare as my DNS service. Deploying their pod with the API token for cloudflare seems to work fine, but the moment I deploy the test pod I start to get a stream of messages that look like this:

level=error msg="error from makeRequest: HTTP request failed: Get \"https://api.cloudflare.com/client/v4/zones?page=1&per_page=50\": x509: certificate is valid for unifi.local, localhost, [::1], not api.cloudflare.com"
level=error msg="error from makeRequest: HTTP request failed: Get \"https://api.cloudflare.com/client/v4/zones?page=1&per_page=50\": x509: certificate is valid for unifi.local, localhost, [::1], not api.cloudflare.com"
level=error msg="error from makeRequest: HTTP request failed: Get \"https://api.cloudflare.com/client/v4/zones?page=1&per_page=50\": x509: certificate is valid for unifi.local, localhost, [::1], not api.cloudflare.com"

And my DNS does not get updated.

I find it peculiar that I"m getting 'unifi.local' - that's my Ubiquiti router/gateway device's certificate it's picking up, but I'm not sure why it's looking for api.cloudflare.com? I'm also not sure what I should do here - the 'unifi.local' certificate probably isn't ideal, but I definitely shouldn't be changing the cert over to appear to be coming from api.cloudflare.com.... right?

Updated information:

As requested by user x4k3p, some more information:

I'm using the standard Core DNS that comes with microk8s. Here's microk8s status output:

microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dashboard            # The Kubernetes dashboard
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    helm3                # Helm 3 - Kubernetes package manager
    ingress              # Ingress controller for external access
    metallb              # Loadbalancer for your Kubernetes cluster
    metrics-server       # K8s Metrics Server for API access to service metrics
    prometheus           # Prometheus operator for monitoring and logging
    registry             # Private image registry exposed on localhost:32000
    storage              # Storage class; allocates storage from host directory
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    host-access          # Allow Pods connecting to Host services smoothly
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    rbac                 # Role-Based Access Control for authorisation

dig output for api.cloudflare.com:

; <<>> DiG 9.16.1-Ubuntu <<>> +trace api.cloudflare.com
;; global options: +cmd
.                       7169    IN      NS      m.root-servers.net.
.                       7169    IN      NS      l.root-servers.net.
.                       7169    IN      NS      k.root-servers.net.
.                       7169    IN      NS      j.root-servers.net.
.                       7169    IN      NS      i.root-servers.net.
.                       7169    IN      NS      h.root-servers.net.
.                       7169    IN      NS      g.root-servers.net.
.                       7169    IN      NS      f.root-servers.net.
.                       7169    IN      NS      e.root-servers.net.
.                       7169    IN      NS      d.root-servers.net.
.                       7169    IN      NS      c.root-servers.net.
.                       7169    IN      NS      b.root-servers.net.
.                       7169    IN      NS      a.root-servers.net.
;; Received 262 bytes from 127.0.0.53#53(127.0.0.53) in 0 ms

com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20210109170000 20201227160000 26116 . CxVQqdDVxRu9qppYx+yycTfWZb9YCwwWzWysdZ+N1d57xjEvxjgdLwlz aNb6LE3AVn90qtba6b+foAVU91kgCVtlGTruj7cuxNbAhrwx+aNPCfXh fAldBiPjAyBjIxi9KmFxv2cZx6+koKvVfY6ZhTydTArq/YhHT2Q87LPr JkRMDTn/qasoLhBLGek0ibkR7l8JwnjLnWhmWR85ZaeIDCpmxQxWzWOR 1rWCRsE3uIAL+UIlcwHIlSQOqH9xPwHtkL+M17+7wJdoS7RkW541nxZP yX8yGWA9KykYcyz/SWV/jDF2gBgZ3Ouyaay9PpC2hn8m7VCfo5Zi6zdc l5wiqQ==
;; Received 1178 bytes from 192.58.128.30#53(j.root-servers.net) in 4 ms

cloudflare.com.         172800  IN      NS      ns3.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns5.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns4.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns6.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns7.cloudflare.com.
cloudflare.com.         86400   IN      DS      2371 13 2 32996839A6D808AFE3EB4A795A0E6A7A39A76FC52FF228B22B76F6D6 3826F2B9
cloudflare.com.         86400   IN      RRSIG   DS 8 2 86400 20210103052149 20201227041149 31510 com. cRdRaMsAfVArkVJjuH3wpQmhJYVZra2roIrgiEdaFQ3guUpl1pDXVoH5 vMT/tj9CXiNvL/hIKfdq+uto9v+YkCYrB+RlwfoMbwJN3IYMnycHpDAQ LWHxLoV/h/jMq20oC8J311hhCd7b/NjL2fiU3UZrmPwhDjG6rp6EGQIO BiHOOk6NXyIKnMwDrtHaTuZMfwxDYpWV271p+AjukUcPkw==
;; Received 820 bytes from 192.31.80.30#53(d.gtld-servers.net) in 4 ms

api.cloudflare.com.     300     IN      A       104.19.193.29
api.cloudflare.com.     300     IN      A       104.19.192.29
api.cloudflare.com.     300     IN      RRSIG   A 13 3 300 20201229054503 20201227034503 34505 api.cloudflare.com. Eo014eDzLog4AsqibERuyJlXQNrFnFkPfWAAbR138ZLyklOxWcvQ0a83 IMNGOXP+jQwwMqijoYUFXAqI5HEppA==
;; Received 193 bytes from 162.159.6.6#53(ns7.cloudflare.com) in 8 ms

(None of these are my IP; I'm in the 50.xxx.xxx.xxx range)

dig to Google:

; <<>> DiG 9.16.1-Ubuntu <<>> +trace google.com
;; global options: +cmd
.                       7062    IN      NS      m.root-servers.net.
.                       7062    IN      NS      l.root-servers.net.
.                       7062    IN      NS      k.root-servers.net.
.                       7062    IN      NS      j.root-servers.net.
.                       7062    IN      NS      i.root-servers.net.
.                       7062    IN      NS      h.root-servers.net.
.                       7062    IN      NS      g.root-servers.net.
.                       7062    IN      NS      f.root-servers.net.
.                       7062    IN      NS      e.root-servers.net.
.                       7062    IN      NS      d.root-servers.net.
.                       7062    IN      NS      c.root-servers.net.
.                       7062    IN      NS      b.root-servers.net.
.                       7062    IN      NS      a.root-servers.net.
;; Received 262 bytes from 127.0.0.53#53(127.0.0.53) in 0 ms

com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20210109170000 20201227160000 26116 . CxVQqdDVxRu9qppYx+yycTfWZb9YCwwWzWysdZ+N1d57xjEvxjgdLwlz aNb6LE3AVn90qtba6b+foAVU91kgCVtlGTruj7cuxNbAhrwx+aNPCfXh fAldBiPjAyBjIxi9KmFxv2cZx6+koKvVfY6ZhTydTArq/YhHT2Q87LPr JkRMDTn/qasoLhBLGek0ibkR7l8JwnjLnWhmWR85ZaeIDCpmxQxWzWOR 1rWCRsE3uIAL+UIlcwHIlSQOqH9xPwHtkL+M17+7wJdoS7RkW541nxZP yX8yGWA9KykYcyz/SWV/jDF2gBgZ3Ouyaay9PpC2hn8m7VCfo5Zi6zdc l5wiqQ==
;; Received 1198 bytes from 192.112.36.4#53(g.root-servers.net) in 67 ms

google.com.             172800  IN      NS      ns2.google.com.
google.com.             172800  IN      NS      ns1.google.com.
google.com.             172800  IN      NS      ns3.google.com.
google.com.             172800  IN      NS      ns4.google.com.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20201231054042 20201224043042 31510 com. WCw2EkmxTyUDl5OH585paZeOpsJGZxYPmMLvxiyH+Q8/vTnogCTRCOiE oalX4/M3cE7w/RSxMXbbtMkDcCmWvhRTBQ4GUbtuJB+0AeXNmkBsGfLU jJl4dWFrXuLq0bgiu8xeKoIvJmV59EkHWq9iaekMiy9uMi1OxwyBPZBH K6IREH5Zv+ox++OAmyxj/Wzb8AesBehtoFaIpZ3i869l8A==
S84BDVKNH5AGDSI7F5J0O3NPRHU0G7JQ.com. 86400 IN NSEC3 1 1 0 - S84CDVS9VPREADFD6KK7PDADH0M6IO8H NS DS RRSIG
S84BDVKNH5AGDSI7F5J0O3NPRHU0G7JQ.com. 86400 IN RRSIG NSEC3 8 2 86400 20210101053343 20201225042343 31510 com. EblXvGFMwJYHdEKosFJGylU+cE8tc4mdFZxDTHGcLI7Ae6aONxgWV/xk V55P+J6W4xsGS78a0OY/6ZQk+b3xLKbXuKaUwxlf8xUgzv+1Qt9JQ6Iz rTayg880COrXePjjFyh62Q9jQTsNgRtDRBSOnxjnwuvy+/BwxA4Tm652 ZqvzRM/DwZlo2X/u3xVwYt9qIHoL4wZtI+DpJAKDI5WgTg==
;; Received 836 bytes from 192.35.51.30#53(f.gtld-servers.net) in 35 ms

google.com.             300     IN      A       172.217.3.174
;; Received 55 bytes from 216.239.34.10#53(ns2.google.com) in 12 ms

Please add additional info like your DNS setup within the cluster, maybe providing such info like `dig +trace` (f.e. using netshoot, better with additional tries to google.com aso.) to see why it resolves to your router — x4k3p, Dec 28 '20 at 04:40
Can you exec into the external-dns pod and do a curl to `curl -v https://api.cloudflare.com/client/v4/zones?page=1&per_page=50`? Maybe curl needs to be installed first (or you use the kubectl debug feature) And, please, provide your external-dns configs without secrets, if possible — x4k3p, Dec 28 '20 at 04:53
Unfortunately curl isn't installed, and my microk8s doesn't seem to support 'kubectl debug'. :( That said, I tried wget and got an error of: `ssl_client: api.cloudflare.com: certificate verification failed: self signed certificate wget: error getting response: Connection reset by peer`. Going to look for the external-dns configs next. — Eddie Parker, Dec 28 '20 at 15:32
For me it looks like there is something wrong in the dns resolution. You can try to explicitly set cloudflare in the dns config to resolve correct https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config but I would try to debug why it resolve it to the router. Do you have any proxy-settings anywhere? Try to inspect the resolv.conf in the pod. — x4k3p, Dec 28 '20 at 22:00

kubernetes external-dns not updating dns: "HTTP request failed: Get [url] : x509: certificate is valid for [router domain], not api.cloudflare.com"

0 Answers0