3

We have an Azure DevOps YAML multi-stage pipeline where code is built and then deployed to a sequence of environments. Deployment is achieved using Terraform.

i.e.

Builds -> Test -> Deploy to DEV -> Deploy to TEST - - ->

This pipeline is used for both CI/CD builds and also PR builds. For the PR build, the only part of the deployment stage is running terraform plan on the TF scripts.

For PRs, the pipeline is configured to cancel the ongoing build if changes are pushed to that PR.

The problem we're seeing is that when changes are pushed to the PR and the ongoing build is cancelled, sometimes that cancellation happens during the terraform plan step. This occasionally means that the blob lease taken by terraform plan is not released. From that point onwards, manual intervention is required (break the blob lease) in order for the deployment stages to run successfully.

I believe we can switch off the setting which causes the ongoing PR build to be cancelled if changes are pushed.

But I wondered if there was a way of marking a pipeline step as critical - i.e. it should run to completion if the build is cancelled?

There are other ways of cancelling a pipeline build and there must be other tasks/steps which should not be cancelled part-way through. Such a critical-task setting would cover these situations too.

Nick
  • 25,026
  • 7
  • 51
  • 83
  • 2
    Can you tell how did you configure cancelling? – Krzysztof Madej Jun 16 '21 at 10:10
  • We're using YAML pipelines so I believe the `autocancel` property on the `pr` trigger is what controls this: https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema#pr-trigger – Nick Jun 16 '21 at 12:19

2 Answers2

2

Not sure if you solved this, but I had the exact same issue. Adding condition: always() to my task forced it to complete when DevOps cancelled the pipeline after additional changes were pushed.

See https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml:

jobs:
- job: Foo

  steps:
  - script: echo Hello!
    condition: always() # this step will always run, even if the pipeline is canceled
Tom
  • 21
  • 2
  • Thanks for your answer. I think my situation is slightly different. I want to prevent an already running task from being cancelled (i.e. the Terraform Plan task) rather than making sure a task is always run regardless of the previous tasks. Or maybe I'm misunderstanding your solution? – Nick Aug 23 '22 at 13:02
1

I'm afraid you won't be able to achieve this using just YAML. What you can do will require some additional effort (and in some cases quite big):

  • replace terraform script with bicep for isntance - for me syntax iq quite similar and what you get here is lack of terrafrom state
  • add in your very first step a check if your state is locked and break a lease if it is needed

I understood that you would like to hear something better than this but at the moment there is no way to mark a task as non-cancellable. However, this sound like a cool feature and candidate for feature request.

Krzysztof Madej
  • 32,704
  • 10
  • 78
  • 107
  • Thanks for the suggestions. Can `bicep` prevent multiple processes attempting to provision the resources at the same time? I guess that's the main reason for the blob lease for the terraform step. Randomly breaking the lease sounds like an interesting (if somewhat risky) strategy! – Nick Jun 16 '21 at 13:23
  • 1
    I don't know how you have designed your deployments but you can consider [exclusive locks per environment](https://learn.microsoft.com/en-us/azure/devops/release-notes/2020/pipelines/sprint-172-update). This may solve potential issue with bicep and concurrent deployments. – Krzysztof Madej Jun 16 '21 at 14:48
  • Thanks for the link - it looks interesting... I'll certainly look to `bicep` in future. We're stuck with `terraform` on this project though, but we can't be the only ones suffering from this cancellation issue!? – Nick Jun 17 '21 at 09:12