0

Dears,

I am very new to Kubernetes and I'm currently working on the update process of my services (traefik, prometheus, ...). I want to avoid the compulsive real-time updates that may lead to bugs or crash. I am used to keep the control about what need to be updated and what does not.

So far, I understood that Kubernetes provides the field spec.updateStrategy.type with 2 possible values:

  • RollingUpdate: permanent auto-update
  • OnDelete: auto-update after the manual deletion of a pod

I am surprised to not find the same steps than with apt Debian tool: when I use apt update; apt upgrade, I get a list of what will be updated and I choose what I want to be updated.

When I came to Kubernetes, I imagined updates would allow to keep this two-steps spirit, something like :

  1. Execute a command to compare the current docker images that are deployed on the cluster to the repos. This command would print the new existing version of each images.
  2. Execute another command to choose what will be updated.

There is no stable, unstable, testing channels like Linux repositories with docker, then I have no way to make difference between the testing update and the trustworthy updates. I am affraid that RollingUpdate would deploy each new image without distinction.

Which lead to my main question: is it completely safe to blindly trust RollingUpdate ?

n0n0bstan
  • 1,790
  • 4
  • 15
  • 26
  • This depends on your application. It must be designed for this way of work, e.g. support n-1 version compatibility. – Jonas Apr 26 '21 at 17:35
  • The question is more about opensource applications that are developed by unknown people with unknown validation process and that I use on my cluster (traefik, prometheus, grafana, ....). – n0n0bstan Apr 26 '21 at 17:39
  • They are different, read their docs. E.g. Traefik is stateless (uses Deployment) while Prometheus is stateful (uses StatefulSet). – Jonas Apr 26 '21 at 17:42

1 Answers1

1

It is safe to trust the RollingUpdate StatefulSet update strategy. However, it is important to note that this is not an "auto-update".

A StatefulSet (and also a Deployment) has a template Pod spec that has an image:. That tends to name a fairly specific version of the image to run. In general Kubernetes will check to see if an image with that name and tag is already on the node, and if so, tries to run it without updating it (see Updating images, which discusses the imagePullPolicy: field).

So if your StatefulSet says image: prom/prometheus:2.26.0, for example, your Pods will use exactly that version of Prometheus, and no other. If you just say image: prom/prometheus with no tag (or with ...:latest) then each node will pull whatever version happens to be current at the time the Pod starts up, and you can easily wind up with out-of-sync versions.

The place the update strategy gets used, then, is when you change image: prom/prometheus:2.27.0. The image in a Pod spec can't be modified, so the existing pods need to be deleted and recreated. If you have updateStrategy: { type: RollingUpdate } (the default) then the StatefulSet controller will do this for you, specifically deleting and recreating the pods in order. If you have it set to OnDelete then you need to manually delete the pods.

Since you as the administrator control the image: then you can pick exactly what version is getting used; there is no such thing as a "stable update channel" that will automatically update. Correspondingly, you can deploy the updated StatefulSet YAML configuration in a pre-production environment to test out before you promote it to production.

David Maze
  • 130,717
  • 29
  • 175
  • 215
  • Thank you very much for this clear answer ! It brings to light many points that I couldn't understand through my readings ! The following question is: how do you keep updated about the new releases of the images that you use ? (do you have notifications when prometheus create a new version/tag ?) – n0n0bstan Apr 26 '21 at 18:22