How to roll back a deployment to a previous revision automatically when deployment has crossed 'progressDeadlineSeconds'?

Question

I'm working on deploying a docker image in kubernetes. The first time I deployed the container, I used:

kubectl apply -f <deployment_file>.yaml

and the container was successfully deployed in a pod.

Also, the deployment_file looks something like this:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: publisher
spec:
  replicas: 2
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 300
  progressDeadlineSeconds: 900
  template:
    metadata:
      labels:
        app: publisher
    spec:
      containers:
      - name: publisher
        image: 123dev.azurecr.io/publisher:{{ci-build-number}}
        env:
          - name: ENVIRONMENT
            value: "dev"
        ports:
        - containerPort: 8080

I have defined the 'progressDeadlineSeconds' attribute in the yaml file above.

To keep a track of the deployment, I used:

kubectl rollout status deployment.v1beta1.apps/publisher

Now if I want to update the container with a new image, I can again reuse the command

kubectl apply -f <deployment_file>.yaml

to apply the update.

But what if applying the update fails due to some reason (let's say the docker image is corrupt), is there a way to automatically trigger a rollback to the previous revision when - pods status is not set to 'running' OR the execution time crosses 'pregressDeadlineSeconds'?

Till now I haven't found a way to execute a rollback automatically. Thoughts would be appreciated.

score 7 · Answer 1 · answered May 24 '19 at 12:02

follow the below steps.

1. append the --record param to deployment, as shown below
kubectl apply -f <deployment_file>.yaml --record

2. kubectl rollout history deploy <deployment-name> to check deployment history. for example,
kubectl rollout history deploy mynginx
deployments "mynginx"
REVISION  CHANGE-CAUSE
3         kubectl set image deploy mynginx mynginx=nginx:1.12.2
4         kubectl set image deploy mynginx mynginx=nginx:1.13.8


3. you can rollback to previous version using revision, say to revision 3
kubectl rollout undo deploy mynginx --to-revision=3

Ben Even Tsur · Accepted Answer · 2020-01-02T16:26:34.477

After the kubectl apply command you can check if the deployment rolled out successfully or not and then, if necessary, the kubectl rollout undo command can rollback to the previous revision. Also, you can use the sleep Linux command to wait some time before that.

On Linux:

sleep 900 && \
if [[ $(timeout 5 kubectl rollout status -f <deployment_file>.yaml) != *"successfully rolled out"* ]]; then     
    kubectl rollout undo -f <deployment_file>.yaml
fi > /dev/null 2>&1 &

timeout 5 stops the kubectl rollout status if its stuck after 5 seconds.

> /dev/null 2>&1 & runs the command in the background and without terminal output.

score 4 · Answer 3 · edited Oct 19 '21 at 12:55

This is the steps which I follow to undo the rollout of a deployment

Here client-depl is the name of the deployment

Step.1 To get the list of history of a specific deployment use this command

kubectl rollout history deploy client-depl

after that you will get output like this.

rollout history deploy image

so let say I want to rollout the deployment to REVISION 1 then use this command

$ kubectl rollout undo deploy client-depl --to-revision=1

after that you will get output like this

deployment rollback image

congratulations you have successfully rollout the deployment to previous version. For any doubt please reply will happy to help you.

How to roll back a deployment to a previous revision automatically when deployment has crossed 'progressDeadlineSeconds'?

3 Answers3