2

I have a GKE cluster running multiple nodes across two zones. My goal is to have a job scheduled to run once a week to run sudo apt-get upgrade to update the system packages. Doing some research I found that GCP provides a tool called "OS patch management" that does exactly that. I tried to use it but the Patch Job execution raised an error informing

Failure reason: Instance is part of a Managed Instance Group.

I also noticed that during the creation of the GKE Node pool, there is an option for enabling "Auto upgrade". But according to its description, it will only upgrade the version of the Kubernetes.

Mauricio
  • 2,552
  • 2
  • 29
  • 43

2 Answers2

2

According to the Blog Exploring container security: the shared responsibility model in GKE:

For GKE, at a high level, we are responsible for protecting:

  • The nodes’ operating system, such as Container-Optimized OS (COS) or Ubuntu. GKE promptly makes any patches to these images available. If you have auto-upgrade enabled, these are automatically deployed. This is the base layer of your container—it’s not the same as the operating system running in your containers.

Conversely, you are responsible for protecting:

  • The nodes that run your workloads. You are responsible for any extra software installed on the nodes, or configuration changes made to the default. You are also responsible for keeping your nodes updated. We provide hardened VM images and configurations by default, manage the containers that are necessary to run GKE, and provide patches for your OS—you’re just responsible for upgrading. If you use node auto-upgrade, it moves the responsibility of upgrading these nodes back to us.

The node auto-upgrade feature DOES patch the OS of your nodes, it does not just upgrade the Kubernetes version.

Gabriel Robledo Ahumada
  • 1,542
  • 2
  • 14
  • Point of clarification: it's not a live patch/not an in place update of host OS (makes sense since most updates would require reboot/changes would get lost), that setting updates the GKE version of the nodepool's MIG to use a newer Ubuntu VM image, the replacement VM would have more vulnerabilities pre-patched. "node auto-upgrades helps you keep the nodes in your cluster up-to-date with the cluster control plane version when your control plane is updated" (source: https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades#overview) – neoakris Dec 14 '22 at 06:37
  • Also interesting is that that's a VM manager OS patch management service that does vuln scans. odd thing is even a fresh copy of ubuntu VM from newest version of rapid release channel will have patches available. So it's odd to see that the ubuntu VMs tend to lag behind on patches more than you'd expect even with important CVEs that have patches available, they just don't seem to be pre-baked into the VM image like you'd expect. I plan to ask support about it soon. – neoakris Dec 14 '22 at 06:40
1

OS Patch Management only works for GCE VM's. Not for GKE

You should refrain from doing OS level upgrades in GKE, that could cause some unexpected behavior (maybe a package get's upgraded and changes something that will mess up the GKE configuration).

You should let GKE auto-upgrade the OS and Kubernetes. Auto-upgrade will upgrade the OS as GKE releases are inter-twined with the OS release.

One easy way to go is to signup your clusters to release channels, this way they get upgraded as often as you want (depending on the channel) and your OS will be patched regularly.

Also you can follow the GKE hardening guide which provide you with step to make sure your GKE clusters are as secured as possible

boredabdel
  • 1,732
  • 3
  • 7