11

I have created a GKE private cluster (version: 1.13.6-gke.13) using the following command:

gcloud container clusters create a-cluster-with-user-pass \
 --network vpc-name \
 --subnetwork subnet-name    \
 --enable-master-authorized-networks \
 --username random \
 --password averylongpassword \
 --enable-ip-alias \
 --enable-private-nodes \
 --enable-private-endpoint \
 --master-ipv4-cidr xxx.xx.xx.xx/28 \
 --cluster-version 1.13.6-gke.13 \
 --num-nodes 2 \
 --zone asia-south1-a

I can see that the port (10255) is open in both the nodes (or we can say GCP compute instances) created in the above cluster.

If I create a simple GCP compute instances (so I have in total 3 VM instances) and try to access the internal IP of the GKE node on 10255 port from this VM I am able to access it without any authentication or authorization. Below is the command used to create the GCP compute instance:

gcloud compute instances create vm-name \
 --network vpc-name \
 --subnetwork subnet-name    \
 --zone asia-south1-a

If I send a simple CURL GET request to (xxx.xx.xx.xx:10255/pods) I get tons of information about the pods and applications. As I can see in the documentation of Kubernetes here, it is mentioned that:

--read-only-port int32
     The read-only port for the Kubelet to serve on with no authentication/authorization (set to 0 to disable) (default 10255)

I tried disabling the port by editing kube-config.yaml file in the node by doing an ssh and restarting the kubelet and I was successful. But is this a good approach? I believe there could be multiple issues when xxx.xx.xx.xx:10255/metrics is disabled. Is there way to secure the port? Rather than disabling it?

I see this github issue and I am certain that there is a way to secure this port. I'm not sure how to do that.

I see Kubernetes documentation in general provides us with multiple ways to secure the port. How to do that in Google Kubernetes Engine?

Amit Yadav
  • 4,422
  • 5
  • 34
  • 79

3 Answers3

3

Kubelet is exposing the collected node metrics using this port. Failure to expose these metrics there might lead to unexpected behavior as the system will be essentially flying blind.

Since GKE is a managed system, you're not really supposed to tweak the kubelet flags as the settings will be reset when a node gets recreated (nodes are based in GCE templates that will not include your own configuration).

As for security, I think is safe to leave that port as is, since you're using a private cluster, meaning that only the resources in the same VPC are allowed to reach the nodes.

yyyyahir
  • 2,262
  • 1
  • 5
  • 14
  • But is there a way to at least add authentication or authorization to the endpoint so that not all the VMs but only the allowed VMs access the endpoint? – Amit Yadav Jul 07 '19 at 05:24
  • 1
    As suggested in other responses, you can add VPC level firewall rules to disable access. Remember that all the nodes in a GKE cluster have [network tags](https://cloud.google.com/vpc/docs/add-remove-network-tags), that might make the firewall operation easier for you. – yyyyahir Jul 08 '19 at 18:13
2

As Yahir Hernández suggested in his answer that this port is used to expose metrics related to the system that ensures smooth operation. It might not be a good idea to disable this port.

What we need to do is to prevent access to this port from outside the VPC.

Since you are using GKE on GCP. If you are using VPC you can add firewall rules to port (10255) to allow incoming traffic only from the resources on VPC. Disable access to this port from internet.

Ankit Deshpande
  • 3,476
  • 1
  • 29
  • 42
  • If you look closely, cluster that I created only has internal IP address and by default it is not accessible from outside the VPC. In fact, private clusters are not even accessible by the VMs in another Subnetwork in the same VPC. So I have no concern about traffic from outside the VPC or subnets. I'm concerned about traffic from the VMs in the same Subnetwork. These VMs without having any scope or authentication can access this port by simply sending a GET request. I wanna add authentication in that. – Amit Yadav Jul 07 '19 at 09:58
  • ok. what is the problem you are trying to solve ? Why will you have traffic from within the VM's, given that only your pods will be running inside it ? – Ankit Deshpande Jul 07 '19 at 10:18
  • The problem that I am trying to solve is related to security. Say there is a VM in the same subnet create for some other purpose and not to access the clusters. Even without any authorization and authentication, GKE is allowing the VMs to access this port and get a lot of information. – Amit Yadav Jul 07 '19 at 11:04
  • Then shouldn't it be in a different subnet ? If not try blocking outgoing traffic to port 10255 from the VM. – Ankit Deshpande Jul 07 '19 at 11:21
  • I thought of that solution but creating one Subnet just for a cluster seems like overkill to me. And `blocking outgoing traffic to port 10255 from the VM` doesn't really look like a good solution. What if the cluster is recreated and the IP changes? What if there is another cluster in the same subnet which access the other clusters IP:10255? So, instead of restricting the caller, I would prefer securing the Target. – Amit Yadav Jul 07 '19 at 14:19
2

According to the CIS Google Kubernetes Engine (GKE) Benchmark v1.0.0 page 196 and 197, "Recommendations" > "Kubelet":

  • it is recommended (widely applicable, should be applied to almost all environments) to disable the read-only port 10255
  • you can do this by editing the kubelet config file to set readOnlyPort to 0 and then restarting the kubelet service

At the same time, Google mentions (point 4.2.4) that the port is not disabled by default since:

Some GKE monitoring components use the kubelet read-only port to obtain metrics.

The recommendation from the CIS benchmark is tone-deaf and close to worthless.

  • The point of GKE is to not have to manage the kubelets yourself.
  • It's not clear what effects the recommendation will have on GKE's monitoring of your cluster.
  • It's not at all obvious how you would keep the setting permanent in an auto-scaling cluster. (A daemonset running as privileged, with its only purpose to overwrite GKE's kubelet config?)

In my opinion, the best mitigation that you can do is:

  1. Ensure the port is only accessible from inside the VPC.
  2. Set good egress network policies for your pods. (Or in some other way manage your egress traffic.) Avoid allowing pods to allow all egress on all ports.
DrPhil
  • 377
  • 1
  • 12