3

I'm used to using the classic Devops "Release" pipelines for deploying code changes to Kubernetes clusters. Recently I've been looking into switching to using Azure Pipelines "deployment" jobs together with "Environments". It seems to work really well and I like a lot of the features, like being able to inspect the Kubernetes entities associated with your deployments, and track the deployment history.

Something that I'm accustomed to from the classic Release pipelines is rolling back to an old deployment if it is discovered that a bug has been released (to production for example). Since Release pipelines are based on build artifacts, you simply run the deployment on the old artifact in the Releases UI.

Now using deployments under the Environments tab, I'm not sure how to run a rollback, short of actually making a code change to revert back to the old state (and run through CI builds again needlessly). Another option is, since the deployment is done relative to the code (or commit) rather than an artifact, one could manually run a new pipeline and target the given commit - but this is quite cumbersome to achieve in the Devops UI, and seems prone to errors. In my opinion rolling back should be really easy to achieve, and not prone to errors.

Any ideas how to do this? Here is a sample of my yaml file

trigger:
  batch: true
  branches:
    include:
      - master

pr:
  branches:
    include:
      - master

variables:
  azureContainerRegistry: <registryUrl> 
  azureContainerRegistryServiceConnection: <serviceConnection>
  kubernetesConfigPath: kubernetes
  kubernetesNamespace: <my-namespace>
  major: 0
  buildNumber: $(major).$(Build.BuildId)
  imageName: "$(azureContainerRegistry)/<my-app>:$(buildNumber)"


stages:
  - stage: Bake
    displayName: "Build and Push image"
    jobs:
      - job: Validate
        displayName: "Build image"
        pool:
          name: "Docker"
        steps:
          - script: docker build -t $(imageName) .
            displayName: Build App 
      - job: Publish
        displayName: "Push image"
        dependsOn: Validate
        condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
        pool:
          name: "Docker"
        steps:
          - task: Docker@2
            displayName: Login to Container Registry
            inputs:
              command: login
              containerRegistry: $(azureContainerRegistryServiceConnection)
          - script: docker push $(imageName)
            displayName: PUSH $(imageName)
  - stage: DeployTest
    displayName: "Deploy TEST"
    dependsOn: Bake
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
    jobs:
      - deployment: Deploy
        environment: <my-test-env>.$(kubernetesNamespace)
        pool:
          name: "Docker"
        strategy:
          runOnce:
            deploy:
              steps:
                - task: qetza.replacetokens.replacetokens-task.replacetokens@3
                  displayName: "Replace tokens"
                  inputs:
                    targetFiles: $(kubernetesConfigPath)/base/*.yaml
                    escapeType: none
                    tokenPrefix: "{"
                    tokenSuffix: "}"
                - task: Kubernetes@1
                  displayName: "kubectl apply"
                  inputs:
                    namespace: $(kubernetesNamespace)
                    command: apply
                    arguments: -k $(kubernetesConfigPath)/test
                    versionSpec: 1.7.0
                    checkLatest: true
                - task: Kubernetes@1
                  displayName: "kubectl rollout status"
                  inputs:
                    namespace: $(kubernetesNamespace)
                    command: rollout
                    arguments: "status deployments/<my-app>"
                    versionSpec: 1.7.0
                    checkLatest: true
  - stage: DeployProd
    displayName: "Deploy PROD"
    dependsOn: DeployTest
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
    jobs:
      - deployment: Deploy
        environment: <my-prod-env>.$(kubernetesNamespace)
        pool:
          name: "Docker"
        strategy:
          runOnce:
            deploy:
              steps:
                - task: qetza.replacetokens.replacetokens-task.replacetokens@3
                  displayName: "Replace tokens"
                  inputs:
                    targetFiles: $(kubernetesConfigPath)/base/*.yaml
                    escapeType: none
                    tokenPrefix: "{"
                    tokenSuffix: "}"
                - task: Kubernetes@1
                  displayName: "kubectl apply"
                  inputs:
                    namespace: $(kubernetesNamespace)
                    command: apply
                    arguments: -k $(kubernetesConfigPath)/prod
                    versionSpec: 1.7.0
                    checkLatest: true
                - task: Kubernetes@1
                  displayName: "kubectl rollout status"
                  inputs:
                    namespace: $(kubernetesNamespace)
                    command: rollout
                    arguments: "status deployments/<my-app>"
                    versionSpec: 1.7.0
                    checkLatest: true

mchristos
  • 1,487
  • 1
  • 9
  • 24

1 Answers1

0

Redeploying from the older version of the code is the way to do it.

one could manually run a new pipeline and target the given commit - but this is quite cumbersome to achieve in the Devops UI, and seems prone to errors

This is you need a well-organised source control branching and tagging policy. If you previously deployed from branch "releases/release-20212710.01", then deployed from branch "releases/release-20212710.02", you don't need to make any code changes. Rolling back just means selecting the older branch – which still exists, with the same code as before – and deploying.

Vince Bowdren
  • 8,326
  • 3
  • 31
  • 56
  • Interesting. I'm accustomed to only deploying code thats on the "main" branch. I'm not sure how I would manage all these release branches alongside a mainline branch. – mchristos Oct 28 '21 at 06:35
  • With what you're suggesting, would the release branch be the PR branch? – mchristos Oct 28 '21 at 08:21
  • No, different from a PR branch. The PR branch goes _into_ master, but a release branch is taken _from_ master at the time when you want a controlled release cut. – Vince Bowdren Oct 28 '21 at 10:19
  • Branches are not the only way to achieve this; another alternative is to [tag](https://git-scm.com/book/en/v2/Git-Basics-Tagging) master at the point when you are going to release. – Vince Bowdren Oct 28 '21 at 10:20
  • Right, so you would add the tag in the release (deploy) stage of the pipeline? Could you show an example of doing that in the pipeline yaml? – mchristos Oct 28 '21 at 10:48
  • It's easier to do it manually. To do it in a pipeline, you have to POST to the Azure DevOps API: https://learn.microsoft.com/en-us/rest/api/azure/devops/git/annotated-tags/create?view=azure-devops-rest-6.0 – Vince Bowdren Oct 28 '21 at 11:47