Kubernetes Cron-Job Slack Notification

Question

I have a cronjob which creates postgres backup job.. I would like to send notification to slack channels via webhook with cronjob status fail or success. How can I add a condition or specify the status of Job and sending to slack? I suppose that also below curl request will work but please warn if you see any fault.

kind: CronJob
metadata:
  name: standup
spec:
  schedule: "* 17 * * 1-5"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: standup
            image: busybox
            resources:
              requests:
                cpu: 1m
                memory: 100Mi
            env:
              - args: /bin/sh
              - -c 
              - curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://hooks.slack.com/services/TQPCENFHP/
      restartPolicy: OnFailure

~ semural$ kubectl logs $pods -n database
The following backups are available in specified backup path:
Added `s3` successfully.
[2020-04-13 14:24:46 UTC]      0B postgresql-cluster/


NAME                                SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
postgresql-postgresql-helm-backup   0 0 * * *   False     0        8h              18h

NAME                                           COMPLETIONS   DURATION   AGE
postgresql-postgresql-helm-backup-1586822400   1/1           37s        8h
postgresql-postgresql-helm-backup-list         1/1           2s         18h
postgresql-postgresql-helm-pgmon               1/1           49s        18h

@Kartoch, I updated my question.. Not sure how to get logs from cronjob — semural, Apr 13 '20 at 14:01
Job and CronJob start pods to execute the pod spec, search for the pods prefixed with the name of your CronJob and get their logs — Kartoch, Apr 13 '20 at 14:04
This is not the good log, please read https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/ to understand how job, cronjob and associated pods are named — Kartoch, Apr 13 '20 at 15:45
They are not the good logs. For instance your CronJob is defined in the default namesapce, where on the other side you've put the logs a pod $pods in the namespace dataabse. — Kartoch, Apr 14 '20 at 08:06
Could you put the results of `kubectl describe cronjobs standup` ? — Kartoch, Apr 14 '20 at 08:07

irvifa · Accepted Answer · 2020-04-16T07:05:12.570

I think we can create a simple script to get the cronjob status:

import json
import os
from kubernetes import client, config, utils
from kubernetes.client.rest import ApiException
from api.exceptions import BatchApiNamespaceNotExistedException

class Constants:
    BACKOFF_LIMIT = 1
    STATUS_RUNNING = "RUNNING"
    STATUS_SUCCEED = "SUCCEED"
    STATUS_FAILED = "FAILED"
    STATUS_NOT_FOUND = "NOT FOUND"

class KubernetesApi:
    def __init__(self):
        try:
            config.load_incluster_config()
        except:
            config.load_kube_config()
        self.configuration = client.Configuration()
        self.api_instance = client.BatchV1Api(client.ApiClient(self.configuration))
        self.api_instance_v1_beta = client.BatchV1beta1Api(client.ApiClient(self.configuration))

    def get_job_status(self, job):
        if job is not None:
            total_failed_pod = job.status.failed or 0
            total_succeeded_pod = job.status.succeeded or 0
            if total_failed_pod + total_succeeded_pod < Constants.BACKOFF_LIMIT:
                return Constants.STATUS_RUNNING
            elif total_succeeded_pod > 0:
                return Constants.STATUS_SUCCEED
            return Constants.STATUS_FAILED
        return Constants.STATUS_NOT_FOUND

    def get_cron_job_status(self, namespace):
        try:
            cron_job_list = self.api_instance_v1_beta.list_namespaced_cron_job(namespace=namespace,
                                                                          watch=False)
        except ApiException as e:
            raise BatchApiNamespaceNotExistedException("Exception when calling BatchV1Api->list_namespaced_cron_job: %s\n" % e)

        for cron_job in cron_job_list.items:
          if cron_job.status.active is not None:
            for active_cron_job in cron_job.status.active:
              job = self.api_instance.read_namespaced_job(namespace=namespace,
                                                       name=active_cron_job.name)
              if job_status == Constants.STATUS_FAILED:
                # Do whatever you want in there
                print(job_status)

So if the status is failed then we can send the log to slack.

thanks for your response..it might be a simple questions but first where should i store this script and second how should I trigger this script? — semural, Apr 15 '20 at 14:26
You can create another deployment containing this script, and do while true to keep checking if there's a job failure. — irvifa, Apr 15 '20 at 15:27
@semural I put the code in here: https://github.com/irvifa/kubernetes-batch-api-example — irvifa, Apr 16 '20 at 07:09

score 0 · Answer 2 · answered Sep 09 '21 at 14:16

I think what you have is already a good start.. Assuming you have the curl command as a script that takes the first argument as the message to be posted you can do something as the following:

kind: CronJob
metadata:
  name: standup
spec:
  schedule: "* 17 * * 1-5"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: standup
            image: busybox
            resources:
              requests:
                cpu: 1m
                memory: 100Mi
            env:
              - args: /bin/sh
              - -c 
              - run-job.py || notify-cron-job "FAIL" && notify-cron-job "SUCCESS"

Kubernetes Cron-Job Slack Notification

2 Answers2