I was moving Kubernetes installation from old work infra to my homelab. In the process of movement, etcd snapshot was deployed on homelab, I've changed all certificates and etcd encryption key. Now apparently my cluster can't create secrets for serviceaccounts and so I can't use deployments either. What I've found so far is namespaces don't have secrets associated with serviceaccounts
NAMESPACE NAME SECRETS AGE
default default 0 29d
kube-node-lease default 1 719d
kube-public default 1 719d
kube-system attachdetach-controller 1 719d
kube-system certificate-controller 1 719d
kube-system clusterrole-aggregation-controller 1 719d
kube-system coredns 1 717d
kube-system cronjob-controller 1 719d
kube-system daemon-set-controller 1 719d
kube-system default 1 719d
kube-system deployment-controller 1 719d
kube-system disruption-controller 1 719d
kube-system endpoint-controller 1 719d
kube-system endpointslice-controller 0 294d
kube-system endpointslicemirroring-controller 0 294d
kube-system expand-controller 1 719d
kube-system generic-garbage-collector 1 719d
kube-system horizontal-pod-autoscaler 1 719d
kube-system job-controller 1 719d
kube-system namespace-controller 1 719d
kube-system node-controller 1 719d
kube-system persistent-volume-binder 1 719d
kube-system pod-garbage-collector 1 719d
kube-system pv-protection-controller 1 719d
kube-system pvc-protection-controller 1 719d
kube-system replicaset-controller 1 719d
kube-system replication-controller 1 719d
kube-system resourcequota-controller 1 719d
kube-system root-ca-cert-publisher 0 294d
kube-system service-account-controller 1 719d
kube-system service-controller 1 719d
kube-system statefulset-controller 1 719d
kube-system ttl-controller 1 719d
kube-dev default 1 690d
Timestamp 294days is about time when I've finished rollout of etcd snapshot in lab. All new resources created afterwards don't contain secrets. Even creation of deployment in namespace containing secret (eg "kube-dev" 690 days old) fails with reason of inability to create API token. API server logs shows following
ar 14 08:52:54 controller32 kube-apiserver[1109873]: I0314 08:52:54.874017 1109873 healthz.go:244] informer-sync check failed: readyz
Mar 14 08:52:54 controller32 kube-apiserver[1109873]: [-]informer-sync failed: 1 informers not started yet: [*v1.Secret]
Mar 14 08:52:54 controller32 kube-apiserver[1109873]: E0314 08:52:54.959463 1109873 cacher.go:419] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/lab": no matching prefix found;reinitializing...
Mar 14 08:52:54 controller32 kube-apiserver[1109873]: I0314 08:52:54.973314 1109873 shared_informer.go:266] stop requested
Mar 14 08:52:54 controller32 kube-apiserver[1109873]: I0314 08:52:54.973845 1109873 healthz.go:244] informer-sync check failed: readyz
Mar 14 08:52:54 controller32 kube-apiserver[1109873]: [-]informer-sync failed: 1 informers not started yet: [*v1.Secret]
If I would try to create new serviceaccount and associate it with serviceaccountkey, it will result in empty entry generated. All help will be appreciated.
