I'm using kubespray 2.14 to deploy a k8s cluster. Most of the configuration is default. I've configured OIDC authentication for kubectl. I'm using Keycloak as a locally deployed auth server. Traffic is secured by a self-signed certificate and the domain keycloak.example.com is resolved by the local dns server. I've added a CoreDNS external zone for the example.com domain.
kube-apiserver uses the host network, so names are not resolved by CoreDNS by default.
Everything works fine as long as I use resolvconf_mode: host_resolvconf. Then the coredns address is added to the hosts /etc/resolv.conf file and kube-apiserver uses CoreDNS to resolve custom domain. But this mode makes my cluster highly unstable. I don't want to go deep into this problem because I spent too mach time on it already.
To fix the stability issue, I went back to the default resolvconf_mode: docker_dns, but then i have a OIDC problem.
oidc authenticator: initializing plugin: Get https://keycloak.example.com/auth/realms/test/.well-known/openid-configuration: dial tcp: lookup keycloak.example.com on 8.8.8.8:53: no such host
kube-apiserver can't resolve keycloak.example.com domain because it queires nameservers from the host (8.8.8.8). I thought it should query docker_dns as it is stated in the kubespray documentation:
https://github.com/kubernetes-sigs/kubespray/blob/master/docs/dns-stack.md
For hostNetwork: true PODs however, k8s will let docker setup DNS settings. Docker containers which are not started/managed by k8s will also use these docker options.
Is there a way to configure kubespray inventory to fix this problem without manually adding a nameserver to each master node? Here is my part of the inventory config:
kube_oidc_url: https://keycloak.example.com/auth/realms/test
kube_oidc_client_id: cluster
resolvconf_mode: docker_dns
upstream_dns_servers:
- 192.168.30.47
coredns_external_zones: &external_zones
- zones:
- example.com:53
nameservers:
- 192.168.30.47
cache: 5
nodelocaldns_external_zones: *external_zones
docker-dns.conf
[Service]
Environment="DOCKER_DNS_OPTIONS=\
--dns 10.233.0.3 --dns 192.168.30.47 --dns 8.8.8.8 \
--dns-search default.svc.cluster.local --dns-search svc.cluster.local \
--dns-opt ndots:2 --dns-opt timeout:2 --dns-opt attempts:2 \
"
