Kubernetes blocking until job stops running

Question

I'd like to use kubectl wait to block in a script until a job or pod I launched has either completed or until it has failed. I wanted to do something like kubectl wait --for=condition=* but I can't find good documentation on what the different condition options are. kubectl wait --for=condition=completed doesn't seem to work for jobs in an error state.

Mark McElroy · Accepted Answer · 2020-07-28T17:42:43.487

Note that currently kubectl wait is marked as an experimental command on the kuberenetes documentation page, so it's likely a feature in development and not finished.
Examples from kubectl wait --help and https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#wait :

Examples:
  # Wait for the pod "busybox1" to contain the status condition of type "Ready".
  kubectl wait --for=condition=Ready pod/busybox1
  
  # Wait for the pod "busybox1" to be deleted, with a timeout of 60s, after having issued the "delete" command.
  kubectl delete pod/busybox1
  kubectl wait --for=delete pod/busybox1 --timeout=60s

The documentation also states the only options for the --for parameter are currently delete and condition=condition-name

It appears the condition field in

kubectl wait --for condition=condition-name pod pod-name

refers to entries in pod.status.conditions.
So we can launch a pod and look at the fields in pod.status.conditions for the kind of conditions we'd like to check.
As an example I had a look at the yaml for one of my pods using kubectl get pod pod-name -o yaml and found the following conditions

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-07-25T21:14:32Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-07-25T21:14:41Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-07-25T21:14:41Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-07-25T21:14:32Z"
    status: "True"
    type: PodScheduled

So we can create pods and block until any of these conditions are met by using:

kubectl wait --for condition=Initialized pod my-pod-name
kubectl wait --for condition=Ready pod my-pod-name
kubectl wait --for condition=ContainersReady pod my-pod-name
kubectl wait --for condition=PodScheduled pod my-pod-name

Similarly for jobs:
I created a job, waited for it to complete, then looked at it's yaml and found the following

status:
  completionTime: "2020-07-28T17:24:09Z"
  conditions:
  - lastProbeTime: "2020-07-28T17:24:09Z"
    lastTransitionTime: "2020-07-28T17:24:09Z"
    status: "True"
    type: Complete
  startTime: "2020-07-28T17:24:06Z"
  succeeded: 1

Therefore we can use the conditions in job.status.conditions, such as

kubectl wait --for condition=Complete job job-name

And we can of course add timeouts if we want to handle cases when the job fails to complete

kubectl wait --for condition=Complete job job-name --timeout 60s

Thanks Mark, If I'm reading you comment right, there's no good way to wait for a job to complete? I see some other stackoverflow posts referencing condition=complete https://stackoverflow.com/questions/53536907/kubectl-wait-for-condition-complete-timeout-30s. But I couldn't find details on how that works and if it counts failures — software_intern, Jul 28 '20 at 17:01
@software_intern I've updated the answer to give an example of waiting for a job to complete. In short, use `kubectl wait --for condition=complete job job-name --timeout 60s`. But you may find more conditions that exist in jobs you create, just check them out with `kubectl get job job-name -o yaml` and look under status.conditions — Mark McElroy, Jul 28 '20 at 17:46

Kubernetes blocking until job stops running

1 Answers1