How to roll kubernetes updates in intervals

Question

We have a case where we need to make sure that pods in k8s have the latest version possible. What is the best way to accomplish this?

First idea was to kill the pod after some point, knowing that the new ones will come up pulling the latest image. Here is what we found so far. Still don't know how to do it.

Another idea is having rolling-update executed in intervals, like every 5 hours. Is there a way to do this?

pagid · Accepted Answer · 2017-01-31T19:34:24.433

2

As mentioned by @svenwltr using activeDeadlineSeconds is an easy option but comes with the risk of loosing all pods at once. To mitigate that risk I'd use a deployment to manage the pods and their rollout, and configure a small second container along with the actual application. The small helper could be configured like this (following the official docs):

apiVersion: v1
kind: Pod
metadata:
  name: app-liveness
spec:
  containers:
  - name: liveness
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep $(( RANDOM % (3600) + 1800 )); rm -rf /tmp/healthy; sleep 600
    image: gcr.io/google_containers/busybox

    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5

  - name: yourapplication
    imagePullPolicy: Always
    image: nginx:alpine

With this configuration every pod would break randomly within the configured timeframe (here between 30 and 90mins) and that would trigger the start of a new pod. The imagePullPolicy: Always would then make sure that the image is updated during that cycle.

This of course assumes that your application versions are always available under the same name/tag.

edited Jan 31 '17 at 19:34

answered Jan 31 '17 at 19:28

pagid

13,559
11
78
104

thanks pagid, one question though: in your example, when liveness probe fails won’t just the liveness container be restarted? If just one of the containers in a pod fails the liveness probe will all of the containers on that pod be restarted ? – anvarik Feb 01 '17 at 10:28
The Pod is always scheduled as whole - so a failing container will reschedule the entire pod. – pagid Feb 01 '17 at 11:07
@pagid according to this github issue thread, it's not the case that a failing container will reschedule the entire pod: https://github.com/kubernetes/kubernetes/issues/40908 – Jake Feasel Jan 28 '18 at 22:50
@Jake in case the probe fails beyond the `failureThreshold` it should iirc still restart the Pod ( see https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes ) – pagid Jan 29 '18 at 11:32

svenwltr · Answer 2 · 2017-01-31T18:15:22.627

1

To use your linked feature you just have to specify activeDeadlineSeconds in your pods.

Not tested example:

apiVersion: v1
kind: Pod
metadata:
  name: "nginx"
spec:
  activeDeadlineSeconds: 3600
  containers:
  - name: nginx
    image: nginx:alpine
    imagePullPolicy: Always

The downside of this is, that you cannot control when the deadline kicks in. This means it might happen, that all your pods get killed at the same time and the whole service gets offline (that depends on you applications).

edited Jan 31 '17 at 18:15

answered Jan 31 '17 at 18:06

svenwltr

17,002
12
56
68

thanks @svenwltr, great answers. I accepted pagid's since his was the most thorough – anvarik Feb 01 '17 at 10:28

score 1 · Answer 3 · answered Jan 31 '17 at 18:13

Another alternative is to use a deployment and let the controller handle roll outs. To be more specific: If you update the image field in the deployment yaml, it automatically updates every pod. IMO that's the cleanest way, but it has some requirements:

You cannot use the latest tag. The assumption is that a container only needs an update, when the image tag changes.
If an updated happens, you have to update image tag manually, somehow. This might be done by a custom controller which checks for new tags and updates the deployment accordingly. Or this could be triggered by a Continuous Delivery system.

score 0 · Answer 4 · answered Jan 28 '18 at 23:30

I tried using Pagid's solution, but unfortunately my observation and subsequent research indictate that his assertion that a failing container will restart the whole pod is incorrect. It turns out that only the failing container will be restarted, which obviously does not help much when the point is to restart the other containers in the pod at random intervals.

The good news is that I have a solution that seems to work which is based on his answer. Basically, instead of writing to /tmp/healthy, you instead write to a shared volume which each of the containers within the pod have mounted. You also need to add the liveness probe to each of those pods. Here's an example based on the one I am using:

  volumes:
  - name: healthcheck
    emptyDir:
      medium: Memory
  containers:
    - image: alpine:latest
      volumeMounts:
        - mountPath: /healthcheck
          name: healthcheck
      name: alpine
      livenessProbe:
        exec:
          command:
          - cat
          - /healthcheck/healthy
        initialDelaySeconds: 5
        periodSeconds: 5
    - name: liveness
      args:
      - /bin/sh
      - -c
      - touch /healthcheck/healthy; sleep $(( RANDOM % (3600) + 1800 )); rm -rf /healthcheck/healthy; sleep 600
      image: gcr.io/google_containers/busybox
      volumeMounts:
        - mountPath: /healthcheck
          name: healthcheck
      livenessProbe:
        exec:
          command:
          - cat
          - /healthcheck/healthy
        initialDelaySeconds: 5
        periodSeconds: 5

How to roll kubernetes updates in intervals

4 Answers4