I've been following Kelsey Hightower's Kubernetes the Hard Way which walks you through manually setting up a k8s cluster.
This is not running on minikube - it's running on a remote VPS.
I'm on the step where I set up the k8s control plane.
However when trying to run a health check on kube-apiserver I get the following:
$ kubectl cluster-info --kubeconfig admin.kubeconfig
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
I'm not quite sure where to start debugging from here.
Configuration
All 3 k8s services are running:
systemctl status kube-apiserver kube-controller-manager kube-scheduler etcd
# => All 4 return: "active (running)"
The kube-apiserver is configured to start with systemd:
$ cat /etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-apiserver \
--advertise-address=$INTERNAL_IP_REDACTED \
--allow-privileged=true \
--apiserver-count=3 \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/log/audit.log \
--authorization-mode=Node,RBAC \
--bind-address=0.0.0.0 \
--client-ca-file=/var/lib/kubernetes/ca.pem \
--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
--etcd-cafile=/var/lib/kubernetes/ca.pem \
--etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
--etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
--etcd-servers=https://10.240.0.10:2379,https://10.240.0.11:2379,https://10.240.0.12:2379 \
--event-ttl=1h \
--encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
--kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
--kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
--kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
--runtime-config='api/all=true' \
--service-account-key-file=/var/lib/kubernetes/service-account.pem \
--service-account-signing-key-file=/var/lib/kubernetes/service-account-key.pem \
--service-account-issuer=https://$EXTERNAL_IP_REDACTED:6443 \
--service-cluster-ip-range=10.32.0.0/24 \
--service-node-port-range=30000-32767 \
--tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
--tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
The kube-apiserver is definitely running and listening on port :6443
$ lsof -iTCP -sTCP:LISTEN -n -P | grep 6443
kube-apis 989442 root 7u IPv6 9345693 0t0 TCP *:6443 (LISTEN)
Here is the admin.kubeconfig file that is configured to look for the cluster at 127.0.0.1:6443
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tL...
server: https://127.0.0.1:6443
name: kubernetes-the-hard-way
contexts:
- context:
cluster: kubernetes-the-hard-way
user: admin
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: admin
user:
client-certificate-data: LS0tLS1CRU...
client-key-data: LS0tLS1C....
There is an SSL certificate provisioned (created earlier in the Kubernetes tutorial)
$ ls -hlt /var/lib/kubernetes/ca*
-rw------- 1 root root 1.7K Dec 18 00:56 /var/lib/kubernetes/ca-key.pem
-rw-r--r-- 1 root root 1.3K Dec 18 00:56 /var/lib/kubernetes/ca.pem
Finally, NGINX is also configured to redirect port 80 traffic to the health check end point
$ cat /etc/nginx/sites-available/kubernetes.default.svc.cluster.local
server {
listen 80;
server_name kubernetes.default.svc.cluster.local;
location /healthz {
proxy_pass https://127.0.0.1:6443/healthz;
proxy_ssl_trusted_certificate /var/lib/kubernetes/ca.pem;
}
}
As mentioned above, I'm not really seeing what's wrong and I have no idea where else to start investigating.
Thank you!
