Trigger Kubernetes job on secret update

Question

How do you trigger rerunning a job once one of its dependent secrets get updated?

For example, say I have this simple job:

apiVersion: batch/v1
kind: Job
metadata:
  name: job-test
spec:
  template:
    metadata:
      labels:
        app: databricks
    spec:
      containers:
      - name: job-test
        image: alpine
        command:
          - echo
          - hello $(DAN_TEST)
        env:
          - name: DAN_TEST
            valueFrom:
              secretKeyRef:
                name: dan-test
                key: dan-test-1
      restartPolicy: Never
  backoffLimit: 4

Adding this job makes it run and print out the secret, but when the secret is changed, the job is not automatically rerun.

Is there built-in or 3rd party extension resource that can target the secret and the job and trigger a rerun?

Answer 1

I think it's maybe better to use a ConfigMap instead of a Secret, because ConfigMaps are designed to be more easily updated and can be monitored for changes using Kubernetes' built-in watch functionality!

let me update your code again :

apiVersion: batch/v1
kind: Job
metadata:
  name: job-test
  annotations:
    configmap.reloader.stakater.com/reload: "dan-test"
spec:
  template:
    metadata:
      labels:
        app: databricks
    spec:
      containers:
      - name: job-test
        image: alpine
        command:
          - echo
          - hello $(DAN_TEST)
        env:
          - name: DAN_TEST
            valueFrom:
              configMapKeyRef:
                name: dan-test
                key: dan-test-1
      restartPolicy: Never
  backoffLimit: 4

as you can see,I'm using the "configmap.reloader.stakater.com/reload" annotation to tell Kubernetes to monitor the "dan-test" ConfigMap for changes. and about third-party tools, the "Kubernetes Event-Driven Autoscaling" (KEDA) project includes a "Secret" scaler that can trigger a Job based on changes to a Secret or other tools like Argo Workflows also provide similar functionality.

update :
Ok I think to automatically trigger a rerun of a Job when a ConfigMap or Secret it depends on is updated, you can use the cronjob resource instead of the job resource

Let me explain cronjob to you with this example :

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: job-test
spec:
  schedule: "*/1 * * * *" # run every minute
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: job-test
            image: alpine
            command:
              - echo
              - hello $(DAN_TEST)
            env:
            - name: DAN_TEST
              valueFrom:
                configMapKeyRef:
                  name: dan-test
                  key: dan-test-1
          restartPolicy: OnFailure

As you see,restartPolicy is set to OnFailure, which means that the cronjob will automatically rerun the job if it fails, and also to trigger a rerun of the cronjob when the dan-test ConfigMap is updated, you can use the configmap.reloader.stakater.com/reload annotation on the cronjob

Something like this (but be careful if the cronjob is already running when the ConfigMap is updated, the job may be in progress and cannot be interrupted. In this scenario, you will need to delete the job manually to trigger a rerun) :

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: job-test
  annotations:
    configmap.reloader.stakater.com/reload: "dan-test"
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: job-test
            image: alpine
            command:
              - echo
              - hello $(DAN_TEST)
            env:
            - name: DAN_TEST
              valueFrom:
                configMapKeyRef:
                  name: dan-test
                  key: dan-test-1
          restartPolicy: OnFailure

good luck !