0

This unexpectedly happened today after working seamlessly for at least half a year. kubectl calls to all my GKE clusters are returning: Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2021-03-10T15:02:50+02:00 is after 2020-07-24T00:21:03Z

I tried rerunning gcloud auth login and re-retrieving the cluster creds with gcloud container clusters get-credentials... Nothing works.

I tried manually running gcloud config config-helper and this actually shows that the token exipred about an hour ago... But I couldn't understand why. What am I missing?

antweiss
  • 2,789
  • 1
  • 13
  • 12
  • Have you resolved this? Two different authentications are involved here. The (Google-managed) GKE master's cert and a token issued by GCP for your GCP user account. Your user credentials are automatically refreshed (hourly?) by GCP. GCP uses these tokens to authenticate you against GKE clusters too (using OAuth). The GKE master's certificate evidently expired way back in July of last year and you hadn't refreshed|rotated it. That's the issue here. – DazWilkin Mar 16 '21 at 16:46
  • Thanks @DazWilkin, it's weird because I'm the only one getting this. Other team members can connect fine from their laptops, so what am I missing here? also this only started happening this month, and we are way past July 2020... – antweiss Mar 17 '21 at 17:37
  • 1
    Aha! That's relevant info. You did not mention that your colleagues are able to connect to the cluster in your question. Let's check your `KUBECONFIG`. On Linux `${PWD}/.kube/config` or `kubectl config view`. It's a YAML file, you can `less` it. It should exist and contain a list of `clusters` and one should be named for the GKE cluster in question. There should be a `cluster.server` value. If you have `openssl` you should be able to query this for certificates: `openssl s_client -connect [[HOST]]:[[PORT]]` – DazWilkin Mar 17 '21 at 19:02
  • 1
    You may want to verify the values for `cluster` with your one of your colleagues. The `contexts` should include an entry that binds the `cluster.name` and a `user.name` (that also exists) to a `name`. This `context.name` should match the value in `current-context`. – DazWilkin Mar 17 '21 at 19:07
  • Thanks, I do know how kubeconifg works. As mentioned - I freshly recreated the creds by calling `gcloud container clusters get-credentials` so nothing should be stale there. Worth noting also that this is happening for all our gke clusters. (we have 6). At the same time my local and EKS cluster connections work fine. So it does look like something with my gcloud config might be off. But I still don't know what. – antweiss Mar 18 '21 at 10:37
  • And SSL handshake to the server works fine. – antweiss Mar 18 '21 at 10:38

1 Answers1

1

I didn't find the actual reason for this but what fixed it - was removing and then re-installing Google Cloud SDK as described here: https://cloud.google.com/sdk/docs/install

antweiss
  • 2,789
  • 1
  • 13
  • 12