Yesterday, I built a full-featured example which uses Terraform to create a network and a GKE cluster in Google Cloud Platform. The whole thing runs in Vagrant on a CentOS 7 VM and installs both gcloud, kubectl, and helm. I also extended the example to use Helm to install Spinnaker.
The GKE cluster is called gke-test-1. In my documentation I documented getting kubectl setup:
gcloud container clusters get-credentials --region=us-west1 gke-test-1
After this, I was able to use various kubectl commands to get nodes, get pods, get services, and get deployments, as well as all other cluster management commands. I was able to also use Helm to install Tiller and ultimately deploy Spinnaker.
However, today, the same process doesn't work for me. I spun up the network, subnet, GKE cluster, and the node pool, and whenever I try to use commands to get various resoures, I get this response:
[vagrant@katyperry vagrant]$ kubectl get nodes
No resources found.
Error from server (NotAcceptable): unknown (get nodes)
[vagrant@katyperry vagrant]$ kubectl get pods
No resources found.
Error from server (NotAcceptable): unknown (get pods)
[vagrant@katyperry vagrant]$ kubectl get services
No resources found.
Error from server (NotAcceptable): unknown (get services)
[vagrant@katyperry vagrant]$ kubectl get deployments
No resources found.
Error from server (NotAcceptable): unknown (get deployments.extensions)
Interestingly enough, it seems that some command do work:
[vagrant@katyperry vagrant]$ kubectl describe nodes | head
Name: gke-gke-test-1-default-253fb645-scq8
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/fluentd-ds-ready=true
beta.kubernetes.io/instance-type=n1-standard-4
beta.kubernetes.io/os=linux
cloud.google.com/gke-nodepool=default
failure-domain.beta.kubernetes.io/region=us-west1
failure-domain.beta.kubernetes.io/zone=us-west1-b
kubernetes.io/hostname=gke-gke-test-1-default-253fb645-scq8
When I open a shell in Google Cloud console, after running the same login command, I'm able to use kubectl to do all of the above:
naftuli_kay@naftuli-test:~$ gcloud beta container clusters get-credentials gke-test-1 --region us-west1 --project naftuli-test
Fetching cluster endpoint and auth data.
kubeconfig entry generated for gke-test-1.
naftuli_kay@naftuli-test:~$ kubectl get pods
No resources found.
naftuli_kay@naftuli-test:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-gke-test-1-default-253fb645-scq8 Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-253fb645-tfns Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-8bf306fc-n8jz Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-8bf306fc-r0sq Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-aecb57ba-85p4 Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-aecb57ba-n7n3 Ready <none> 40m v1.8.10-gke.0
naftuli_kay@naftuli-test:~$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.64.1 <none> 443/TCP 43m
naftuli_kay@naftuli-test:~$ kubectl get deployments
No resources found.
The only difference I can see is the difference between the kubectl version; Vagrant has the latest version, 1.11.0, and the Google Cloud console has 1.9.7.
I will attempt to downgrade.
Is this a known issue and what, if anything, can I do to work around it?
EDIT: This is reproducible and I can't find a way to prevent it from recurring. I tore down all of my infrastructure and then stood it up again. The Terraform is available here.
After provisioning the resources, I waited until the cluster reported being healthy:
[vagrant@katyperry vagrant]$ gcloud container clusters describe \
--region=us-west1 gke-test-1 | grep -oP '(?<=^status:\s).*'
RUNNING
I then setup my login credentials:
[vagrant@katyperry vagrant]$ gcloud container clusters get-credentials \
--region=us-west1 gke-test-1
I again attempted to get nodes:
[vagrant@katyperry vagrant]$ kubectl get nodes
No resources found.
Error from server (NotAcceptable): unknown (get nodes)
The cluster appears green in the Google Cloud dashboard:
Apparently, this is a reproducible problem, as I'm able to recreate it using the same Terraform and commands.
