0

What is an appropriate Kubernetes livenessProbe command for a background process?

We have a NodeJS process that consumes messages off an SQS queue. Since it's a background job we don't expose any HTTP endpoints and so a liveness command seems to be the more appropriate way to do the liveness check. What would a "good enough" command setup look like that actually checks the process is alive and running properly? Should the NodeJS process touch a file to update its editted time and the liveness check validate that? Examples I've seen online seem disconnected to the actual process, e.g. they check a file exists.

n00b
  • 5,843
  • 11
  • 52
  • 82
  • Hi there. You could have some global state (I know, usually bad) that is the last date of the happy flow. That way if the NodeJS app gets "stuck" somewhere and stops pulling items from the queue or disconnects and doesnt reconnect, the date field stops updating. Then your NodeJS app can host an API like `/health`. For that endpoint you check if that date field is within the last 1 minute or n minutes. If so return HTTP 200 otherwise return a 500. That way the liveness probe can be used to determine if the pod is in a happy state – Justin Tamblyn Nov 16 '20 at 07:22
  • Seems you can reference the [pg_isready](https://github.com/vishnubob/wait-for-it) script, they use `nc` scan the pg port to determine whether the pg is ready. – FakeAlcohol Nov 16 '20 at 09:33

1 Answers1

0

You could use liveness using exec command.

Here is an example:

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5

To perform a probe, the kubelet executes the command cat /tmp/healthy in the target container. If the command succeeds, it returns 0, and the kubelet considers the container to be alive and healthy. If the command returns a non-zero value, the kubelet kills the container and restarts it.

Vishrant
  • 15,456
  • 11
  • 71
  • 120
  • 1
    Sorry Vishrant, this is the example I was referring to when I said it seems disconnected to a running process. If the container was running a NodeJS process, or a Golang process etc is there a conventional pattern used to check that the process is actually alive and well? – n00b Nov 16 '20 at 01:55