Azure DevOps pipeline, how to write the condition for a stage template to run it for different environments

Question

I have three stages, Dev, QA and Prod and I need to execute the same set of exactly same jobs for all three stages. I have made a stage template and I am calling it from my main Azure DevOps pipeline.

Depending on the build branch, I need to deploy either to only Dev, or to all three environments, Dev, QA and Prod. The issue is I don't know how to set this condition in the stage template.

Here is how the stage template looks like:

parameters:
- name: deployToDev
  displayName: 'Deploy to Dev?'
  type: boolean
  default: false

- name: deployToQA
  displayName: 'Deploy to QA?'
  type: boolean
  default: false

- name: deployToProd
  displayName: 'Deploy to Prod?'
  type: boolean
  default: false

- name: variableGroup
  displayName: 'Variable Group'
  type: string

- name: stageName
  displayName: 'Stage Name'
  type: string

stages:
- stage: ${{parameters.stageName}}
  condition: and(succeeded(), eq( ${{ parameters.deployToProd}}, false), eq( ${{parameters.deploytoQA}}, false) )  
  displayName: 'Deploy to ${{parameters.stageName}}'
  variables: 
  - group: ${{parameters.variableGroup}}
  dependsOn: Build
  jobs:
  - job: <job-name>
  ...

The main pipeline where I call the stage template (I only show the main parts):

pool: 
  vmImage: ubuntu-20.04

trigger:
  branches:
    include:
      - "feature/ORGTHDATAMA-4177"
      - "main"
    exclude:
      - "release"
  paths:
    include:
      - "core_py/*"

parameters:
- name: deployToDevPipelineLvl
  displayName: 'Deploy to Dev?'
  type: boolean
  default: false

- name: deployToQAPipelineLvl
  displayName: 'Deploy to QA?'
  type: boolean
  default: false

- name: deployToProdPipelineLvl
  displayName: 'Deploy to Prod?'
  type: boolean
  default: false

variables:
  pythonVersion: 3.8
  whlPackageName: py_whl_core
  srcDirectory: core_py/$(whlPackageName)
  ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
    BuildFrom: main
    PackageName: $(whlPackageName)
    versionOption: custom
    parameters.deployToDevPipelineLvl: true
    parameters.deployToQAPipelineLvl: true
    parameters.deployToProdPipelineLvl: true
  ${{ else }}:
    BuildFrom: branch_${{ lower(variables['Build.SourceBranchName'] ) }}
    PackageName: $(whlPackageName)_$(BuildFrom)
    versionOption: patch
    parameters.deployToDevPipelineLvl: true
    parameters.deployToQAPipelineLvl: false
    parameters.deployToProdPipelineLvl: false

name: py_whl_build_$(Date:yyyyMMdd)_$(BuildFrom)

stages:
  - stage: Build
    displayName: 'Build Stage'

    jobs:
    - job: <job-name>
    ...


  - template: templates/azure-pipeline-stage-template-us4177.yaml
    parameters:
      deployToDev: ${{parameters.deployToDevPipelineLvl}}
      deployToQA: ${{parameters.deployToQAPipelineLvl}}
      deployToProd: ${{parameters.deployToProdPipelineLvl}}
      variableGroup: databricks-sp-vg-dev-4177
      stageName: DeployToDev

  - template: templates/azure-pipeline-stage-template-us4177.yaml
    parameters:
      deployToDev: ${{parameters.deployToDevPipelineLvl}}
      deployToQA: ${{parameters.deployToQAPipelineLvl}}
      deployToProd: ${{parameters.deployToProdPipelineLvl}}
      variableGroup: databricks-sp-vg-qa-4177
      stageName: 'DeployToQA'

  - template: templates/azure-pipeline-stage-template-us4177.yaml
    parameters:
      deployToDev: ${{parameters.deployToDevPipelineLvl}}
      deployToQA: ${{parameters.deployToQAPipelineLvl}}
      deployToProd: ${{parameters.deployToProdPipelineLvl}}
      variableGroup: databricks-sp-vg-prod-4177
      stageName: DeployToProd

As shown above, I am passing parameters from main pipeline to stage template.These parameters value varies depending on build source branch; I am checking the parameter values in stage template to decide for what environment it runs. The thing is when I run the pipeline from non-main branches the stage template runs for all three environments (Dev, QA and Prod) which is due to the condition.

Questions:

How to correct the condition that it only runs for Dev if the build branch is not main? And how to run for all three environments, when the build branch is main?
Is there a better way to set the parameters deployToDev, deployToQA, and deployToProd? Like having one parameter instead of three?
Should the condition check be in stage at all or should it be in the main pipeline? If it is the better approach how should I implement it?

Krzysztof Madej · Accepted Answer · 2022-09-13T13:39:28.260

I run this:

        - pwsh: |
            echo "$(parameters.deployToDevPipelineLvl) - ${{parameters.deployToDevPipelineLvl}}"
            echo "$(parameters.deployToQAPipelineLvl) - ${{parameters.deployToDevPipelineLvl}}"
            echo "$(parameters.deployToProdPipelineLvl) - ${{parameters.deployToDevPipelineLvl}}"

on the main branch and I got:

true - False
true - False
true - False

And when I ran this on the non-main branch I got:

true - False
false - False
false - False

So your issue here is that you pass runtime parameters:

parameters:
- name: deployToDevPipelineLvl
  displayName: 'Deploy to Dev?'
  type: boolean
  default: false

then you set variables:

  ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
    BuildFrom: main
    PackageName: $(whlPackageName)
    versionOption: custom
    parameters.deployToDevPipelineLvl: true
    parameters.deployToQAPipelineLvl: true
    parameters.deployToProdPipelineLvl: true
  ${{ else }}:
    BuildFrom: branch_${{ lower(variables['Build.SourceBranchName'] ) }}
    PackageName: $(whlPackageName)_$(BuildFrom)
    versionOption: patch
    parameters.deployToDevPipelineLvl: true
    parameters.deployToQAPipelineLvl: false
    parameters.deployToProdPipelineLvl: false

(this will not overwrite parameters).

To pass later parameters as template parameters:

  - template: templates/azure-pipeline-stage-template-us4177.yaml
    parameters:
      deployToDev: ${{parameters.deployToDevPipelineLvl}}
      deployToQA: ${{parameters.deployToQAPipelineLvl}}
      deployToProd: ${{parameters.deployToProdPipelineLvl}}
      variableGroup: databricks-sp-vg-qa-4177
      stageName: 'DeployToQA'

If you just want to run stages based on the branch you don't need all runtime parameters at the main pipeline file.

So with this pipeline:

pool: 
  vmImage: ubuntu-20.04

trigger:
  branches:
    include:
      - "feature/ORGTHDATAMA-4177"
      - "main"
    exclude:
      - "release"
  paths:
    include:
      - "core_py/*"

variables:
  pythonVersion: 3.8
  whlPackageName: py_whl_core
  srcDirectory: core_py/$(whlPackageName)
  ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
    BuildFrom: main
    PackageName: $(whlPackageName)
    versionOption: custom
    deployToDevPipelineLvl: True
    deployToQAPipelineLvl: True
    deployToProdPipelineLvl: True
  ${{ else }}:
    BuildFrom: branch_${{ lower(variables['Build.SourceBranchName'] ) }}
    PackageName: $(whlPackageName)_$(BuildFrom)
    versionOption: patch
    deployToDevPipelineLvl: True
    deployToQAPipelineLvl: False
    deployToProdPipelineLvl: False

name: py_whl_build_$(Date:yyyyMMdd)_$(BuildFrom)

stages:
  - stage: Build
    displayName: 'Build Stage'

    jobs:
    - job: SomeJobe
      steps:
        - pwsh: |
            echo "$(deployToDevPipelineLvl) - ${{variables.deployToDevPipelineLvl}}"
            echo "$(deployToQAPipelineLvl) - ${{variables.deployToQAPipelineLvl}}"
            echo "$(deployToProdPipelineLvl) - ${{variables.deployToProdPipelineLvl}}"

            
  - template: template-file2.yml
    parameters:
      deploy: ${{variables.deployToDevPipelineLvl}}
      variableGroup: databricks-sp-vg-dev-4177
      stageName: DeployToDev

  - template: template-file2.yml
    parameters:
      deploy: ${{variables.deployToQAPipelineLvl}}
      variableGroup: databricks-sp-vg-qa-4177
      stageName: 'DeployToQA'

  - template: template-file2.yml
    parameters:
      deploy: ${{variables.deployToProdPipelineLvl}}
      variableGroup: databricks-sp-vg-prod-4177
      stageName: DeployToProd

and this template:

parameters:
- name: deploy
  displayName: 'Deploy'
  type: boolean
  default: false

- name: variableGroup
  displayName: 'Variable Group'
  type: string

- name: stageName
  displayName: 'Stage Name'
  type: string

stages:
- stage: ${{parameters.stageName}}
  condition: and(succeeded(), eq( ${{ parameters.deploy}}, true))  
  displayName: 'Deploy to ${{parameters.stageName}}'
  dependsOn: Build
  jobs:
  - job: Build
    steps:
    - pwsh: |
        echo "${{ parameters.deploy}}"

You will get on the main branch:

and this on non-main branch:

So you hot this pipeline without parameters at all.

Conditions on the template are good as having them on a higher level is impossible. You could have here if expression, but then you will change the structure of the pipeline (I mean you will not see skipped stages at all). Is it all up to you and your preference?

If you want to keep your runtime parameters:

parameters:
- name: deployToDevPipelineLvl
  displayName: 'Deploy to Dev?'
  type: boolean
  default: false

- name: deployToQAPipelineLvl
  displayName: 'Deploy to QA?'
  type: boolean
  default: false

- name: deployToProdPipelineLvl
  displayName: 'Deploy to Prod?'
  type: boolean
  default: false

pool: 
  vmImage: ubuntu-20.04

trigger:
  branches:
    include:
      - "feature/ORGTHDATAMA-4177"
      - "main"
    exclude:
      - "release"
  paths:
    include:
      - "core_py/*"

variables:
  pythonVersion: 3.8
  whlPackageName: py_whl_core
  srcDirectory: core_py/$(whlPackageName)
  ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
    BuildFrom: main
    PackageName: $(whlPackageName)
    versionOption: custom
    deployToDevPipelineLvl: True
    deployToQAPipelineLvl: True
    deployToProdPipelineLvl: True
  ${{ else }}:
    BuildFrom: branch_${{ lower(variables['Build.SourceBranchName'] ) }}
    PackageName: $(whlPackageName)_$(BuildFrom)
    versionOption: patch
    deployToDevPipelineLvl: True
    deployToQAPipelineLvl: False
    deployToProdPipelineLvl: False

name: py_whl_build_$(Date:yyyyMMdd)_$(BuildFrom)

stages:
  - stage: Build
    displayName: 'Build Stage'

    jobs:
    - job: SomeJobe
      steps:
        - pwsh: |
            echo "$(deployToDevPipelineLvl) - ${{variables.deployToDevPipelineLvl}}"
            echo "$(deployToQAPipelineLvl) - ${{variables.deployToQAPipelineLvl}}"
            echo "$(deployToProdPipelineLvl) - ${{variables.deployToProdPipelineLvl}}"

            
  - template: template-file2.yml
    parameters:
      deploy: ${{ or(variables.deployToDevPipelineLvl, parameters.deployToDevPipelineLvl) }}
      variableGroup: databricks-sp-vg-dev-4177
      stageName: DeployToDev

  - template: template-file2.yml
    parameters:
      deploy: ${{ or(variables.deployToQAPipelineLvl, parameters.deployToQAPipelineLvl) }}
      variableGroup: databricks-sp-vg-qa-4177
      stageName: 'DeployToQA'

  - template: template-file2.yml
    parameters:
      deploy: ${{ or(variables.deployToProdPipelineLvl, parameters.deployToProdPipelineLvl) }}
      variableGroup: databricks-sp-vg-prod-4177
      stageName: DeployToProd

you need test your values like this ${{ or(variables.deployToProdPipelineLvl, parameters.deployToProdPipelineLvl) }}

I see the issue. Is there a way that I still have the runtime parameters and resolve the issue of their values be overwritten in the pipeline? Keeping the runtime parameters to allow pipeline users be able to run the stage according to their need is still a good feature. — E. Erfan, Sep 13 '22 at 09:36
Yes good. But you just mentioned that you want to do it based on the branch or I missed it. If you wan to keep your runtime parameters that's fine. Then just pass them in or condition to the template. You can't overwrite values. — Krzysztof Madej, Sep 13 '22 at 13:34

Azure DevOps pipeline, how to write the condition for a stage template to run it for different environments

1 Answers1