Incorrect liveness probe for Redis not failing

Question

I have configured a liveness probe for my Redis instances that makes sure that the Redis is able to retrieve keys for it to be able to be called 'alive'.

  livenessProbe:
    initialDelaySeconds: 20
    periodSeconds: 10
    exec:
      command:
        {{- include "liveness_probe" . | nindent 16 }}

_liveness.tpl

{{/* Liveness probe script. */}}
{{- define "liveness_probe" -}}

- "redis-cli"
- "set"
- "liveness_test_key"
- "\"SUCCESS\""
- "&&"
- "redis-cli"
- "get"
- "liveness_test_key"
- "|"
- "awk"
- "'$1 != \"SUCCESS\" {exit 1}'"
{{- end }}

The pod is able to start after doing the change. However, I would like to make sure that the probe is working as expected. For that I just added a delete command before the get command.

{{/* Liveness probe script. */}}
{{- define "liveness_probe" -}}

- "redis-cli"
- "set"
- "liveness_test_key"
- "\"SUCCESS\""
- "&&"
- "redis-cli"
- "del"
- "liveness_test_key"
- "&&"
- "redis-cli"
- "get"
- "liveness_test_key"
- "|"
- "awk"
- "'$1 != \"SUCCESS\" {exit 1}'"
{{- end }}

I get the expected exit codes when I execute this command directly in my command prompt.

But the thing is that my pod is still able to start.

Is the liveness probe command I am using okay? If so, how do I verify this?

score 2 · Accepted Answer · answered Jul 09 '21 at 14:31

Try this for your liveness probe it is working fine and you can try the same in readinessProbe:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: redis
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis
    spec:
      containers:
      - image: redis
        name: redis
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - |            
                #!/usr/bin/env bash -e
                #export REDISCLI_AUTH="$REDIS_PASSWORD"

                set_response=$(
                  redis-cli set liveness_test_key "SUCCESS"
                )

                del_response=$(
                    redis-cli del liveness_test_key
                )

                response=$(
                  redis-cli get liveness_test_key
                )

                if [ "$response" != "SUCCESS" ] ; then
                  echo "Unable to get keys, something is wrong"
                  exit 1
                fi               

          initialDelaySeconds: 5
          periodSeconds: 5
          
status: {}

You will need to edit these values in your template

score 1 · Answer 2 · answered Jul 09 '21 at 09:35

1

I think you're confusing livenessProbewith readinessProbe. livenessProbe tells kubernetes to restart your pod if your command returns a non-zero exit code, this is executed after the period specified in initialDelaySeconds: 20

Whereas readinessProbe is what decides whether a pod is in Ready state to accept traffict or not.

  readinessProbe:
    initialDelaySeconds: 20
    periodSeconds: 10
    exec:
      command:
        {{- include "liveness_probe" . | nindent 16 }}

They can also be used together if you need so.

Please check this page from kubernetes documentation where they explain livenessProbe, readinessProbe and startupProbe

answered Jul 09 '21 at 09:35

Yayotrón

1,759
16
27

But in my configuration, shouldn't the liveness probe fail and cause the pod to be restarted? – rishav Jul 09 '21 at 10:02
It should, after 20 seconds, if your command returns a non-zero exit code. I suggest you run kubectl exec into the pod and try running the command from there to be sure it's a non 0 return. – Yayotrón Jul 09 '21 at 10:04
Yup, i already verified this. It does return 1 inside the pod. – rishav Jul 09 '21 at 10:09

Incorrect liveness probe for Redis not failing

2 Answers2