0

we have a batch data processing script inside a container and want to check it is alive and actually working or whether it should be restarted.

It is PHP command line script and doesn't expose any kind of server. Currently running in Docker soon in Kubernetes.

How can we monitor liveness of such script without introducing unnecessary features/libraries?

Jindra
  • 780
  • 13
  • 39
  • 1
    Think how you would manually monitor if it was working and then look to 'automate' that. Remember that probes can be HTTP URLs or execute shell commands on the container. Suppose your batch job was taking an input file and writing to an output file. You might manually look to see if the output file was updated 'recently'. To automate that, you could write a shell script to check the last modified date and exit successfully is recently updated, exit unsuccessfully otherwise. Without knowing more about what your job is doing it is hard to be specific as to what you can do as a probe. – JohnXF Jul 25 '22 at 11:14
  • Thank you for commenting. It is periodically taking data from relational database, denormalizing and puting into a specialized NoSQL database. Manually we check content of the database, but that can be done in ~30 minut intervals. Which is a lot. – Jindra Jul 26 '22 at 10:33
  • You container should only run as long as the PHP script is running - so if the script finishes the container should stop. So in that scenario 'liveness' is really just checking the container is still running. So I would suggest making the PHP script the container command is a good start and should even avoid the need for probes. You could also consider running the container as a cron job - if there is something to do it can do it, if not then it can do nothing. You could schedule it to run as frequently as you think is necessary. – JohnXF Jul 27 '22 at 08:20
  • What you write is true but slightly different. There is a reason Kubernetes and others add probes. The fact that process is running is too basic. We need to detect if it actually is working i.e. isn't stuck in an infinite loop. – Jindra Aug 03 '22 at 05:42

1 Answers1

1

You can use liveness probe command to check whether process is alive or not. Below the sample from k8s documentation. You can write small script to return 0 if process is alive or return non zero code in case of process is dead

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5
Nataraj Medayhal
  • 980
  • 1
  • 2
  • 12