0

Recently the certificates of the Kubernetes cluster have been upgraded. This is an external ETCD managed cluster. Renewed the Master nodes and Worker nodes via Kubeadm upgrade. The external ETCD nodes were upgrade via Kubeadm certs renewal. Afterwards copied the apiserver-etcd-client.crt apiserver-etcd-client.key to the control nodes

ETCD cert check :

CERTIFICATE                          EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
!MISSING! admin.conf
!MISSING! apiserver
apiserver-etcd-client                Oct 11, 2022 20:18 UTC   341d            etcd-ca                 no
!MISSING! apiserver-kubelet-client
!MISSING! controller-manager.conf
etcd-healthcheck-client              Oct 11, 2022 20:29 UTC   341d            etcd-ca                 no
etcd-peer                            Oct 11, 2022 20:29 UTC   341d            etcd-ca                 no
etcd-server                          Oct 11, 2022 20:29 UTC   341d            etcd-ca                 no
!MISSING! front-proxy-client
!MISSING! scheduler.conf

CERTIFICATE AUTHORITY      EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
!MISSING! ca
etcd-ca                    Oct 25, 2029 18:05 UTC   7y              no

Control Plane cert check :

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 11, 2022 20:07 UTC   341d                                    no
apiserver                  Oct 11, 2022 20:07 UTC   341d            ca                      no
apiserver-kubelet-client   Oct 11, 2022 20:07 UTC   341d            ca                      no
controller-manager.conf    Oct 11, 2022 20:07 UTC   341d                                    no
front-proxy-client         Oct 11, 2022 20:07 UTC   341d            front-proxy-ca          no
scheduler.conf             Oct 11, 2022 20:07 UTC   341d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 25, 2029 19:27 UTC   7y              no
front-proxy-ca          Oct 25, 2029 19:27 UTC   7y              no

But all the Kube-API pods on the control nodes generates the following error log. Currently there are no effect to the cluster admin (kubectl commands, scheduling and etc.)

{"log":"E1104 03:25:02.294264       1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is notyet valid\n","stream":"stderr","time":"2021-11-04T03:25:02.294433099Z"}
{"log":"E1104 03:25:02.312771       1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is notyet valid\n","stream":"stderr","time":"2021-11-04T03:25:02.312993805Z"}
{"log":"E1104 03:25:02.332196       1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is notyet valid\n","stream":"stderr","time":"2021-11-04T03:25:02.332386139Z"}
{"log":"E1104 03:25:02.343118       1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is notyet valid\n","stream":"stderr","time":"2021-11-04T03:25:02.343341123Z"}
{"log":"E1104 03:25:02.351442       1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is notyet valid\n","stream":"stderr","time":"2021-11-04T03:25:02.351630063Z"}

Restarted the Kube-API from docker container and checked for any abnormalities on the kube-metrics server and other monitoring services. Yet no traces found

What could be the reason the Kube-API generates such an error?

Dinuka Kavinda
  • 45
  • 1
  • 11
  • This is probably a red herring, but please check if the new certificates have proper dates. You can do this with `openssl x509 -startdate -noout -in ` for start date, and `openssl x509 -enddate -noout -in ` for expiration date. –  Nov 04 '21 at 10:15
  • @p10l Checked all the certificates but they are in-order. Currently checking whether if any external applications are using an old certificate – Dinuka Kavinda Nov 12 '21 at 06:19
  • @WytrzymałyWiktor no still trying to figure out the issue – Dinuka Kavinda Nov 12 '21 at 06:19
  • @DinukaKavinda, any updates? Additionally, did you restart control plane pods after renewing certificates? –  Nov 23 '21 at 12:15

0 Answers0