Turn on/off liveness probe using configMap UI on Rancher

Question

I'm looking for a approach where we can turn on/off liveness probe using configMap UI on Rancher, and same time it does not require redeployment of the pod.

In details : Assume the liveness probe is already been config on pod. Next I have a boolean flag and it is set false, so if false, liveness probe will stop probing on pod. At other time, if set boolean flag to true, liveness probe will resume probe. All these need to works without pod redeployment. Below is my draft idea:

There's a configMap UI (configMap listed on Rancher) which hold the boolean flag to turn on/off liveness probe something like this :

app.liveness.probe.mode=false

Next I want to absorb the above boolean flag into deployment.yaml with the conditional checking as below :

{{ if app.liveness.probe.mode }}
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 70
            periodSeconds: 10
{{ end }}

I'm not sure how to reference the configMap boolean flag on Rancher into the deployment.yaml file.
Or is there any other approach to control the liveness probe on/off toggle without redeployment of pod.

you have to restart the pod or redeployment to new changes take place from deployment.yaml file. — Harsh Manvar, Feb 11 '21 at 03:48
@HarshManvar Thanks. Any other approach to control the liveness probe on/off without modifying the deployment.yaml file? — Tariv, Feb 11 '21 at 05:13

score 0 · Answer 1 · answered Feb 11 '21 at 08:05

Sounds to me as if you want to temporarily switch off you Liveness probe in order to keep your Pod running, even if the Pod is technically not healthy.

Have you considered using a Readiness probe for this?

So, you remove your Liveness probe, which basically keeps your Pod running as long as your application is running.
Your define a Readiness probe that executes HTTP checks on an application endpoint. While that endpoint returns 200, your Pod receives traffic. If you want to take traffic off the Pod, while keeping your Pod running, your readiness probe will have to return an error code (>=400).
To accomplish this dynamically with ConfigMaps, you could mount a ConfigMap as a volume into your application container putting a config file onto your container file system. The config file contains a flag like readninessProbe.active=true. This file is read by your Readiness probe endpoint and returns a 200 or 500 dependent on the value of the boolean.

Naturally, a similar approach could be used to control your Livness probe endpoint, but the K8s mechanism for keeping Pods running but taking traffic off them is taken care of by Readiness probes.

Basically I want stop triggering Liveness probe endpoint. Is this do able? — Tariv, Feb 11 '21 at 14:42
Not out of the box. If your application process keeps running, you can mute your health endpoint programmatically, e.g. stop checking the database along the lines that I have outlined for the readiness probe. — Fritz Duchardt, Feb 12 '21 at 08:20

Turn on/off liveness probe using configMap UI on Rancher

1 Answers1