0

I have an Azure DevOps Multi Stage Pipeline with multiple agents. Generally my stages consist of:

  • Build and Publish Artifacts
  • Deploy to environment 1
  • UI Test environment 1
  • Deploy to environment 2
  • UI Test environment 2

and so on for several environments. How do I allow simultaneous pipelines runs (e.g. Building and Publishing Artifacts for Build #2 while simultaneously UI Testing environment 2 for Build #1), while ensuring that no two agents will ever perform a UI Test for a given environment at the same time?

trigger: batch seems close to what I want, but I believe that disallows concurrency at the pipeline level, not at the stage level.

riQQ
  • 9,878
  • 7
  • 49
  • 66
Lee Richardson
  • 8,331
  • 6
  • 42
  • 65

3 Answers3

1

It seems that if somebody runs the release pipeline twice, You need it to run it one by one, not in parallel, right?

As a workaround:

Each release will start from stage 1. Thus we can add a PowerShell task as the first task for stage 1 to check if there are previous in-progress deployments.

In this PowerShell task, we can call this Rest API to check release stage status.

Power shell script:

# Base64-encodes the Personal Access Token (PAT) appropriately
    $token = "$(pat)"
    $base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
    $success = $false
    $count = 0
do{
    try{
    $stageurl2 = "https://vsrm.dev.azure.com/{org name}/{project name}/_apis/release/deployments?definitionId={release definition ID}&deploymentStatus=inProgress&api-version=6.0"
    $stageinfo2 = Invoke-RestMethod -Method Get -ContentType application/json -Uri $stageurl2 -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}
   
    $inprogressdeployments = $stageinfo2.value | where {($_.deploymentStatus -eq "inProgress")  -and ($_.release.name -ne $ENV:RELEASE_RELEASENAME)  -and ($_.releaseEnvironment.name -ne 'stop services')} | Sort-Object -Property completedOn -Descending
    #write-output $inprogressdeployments
    $stageurl3 = "https://vsrm.dev.azure.com/{org name}/{project name}/_apis/release/deployments?definitionId={release definition ID}&operationStatus=QueuedForAgent&api-version=6.0"
    $stageinfo3 = Invoke-RestMethod -Method Get -ContentType application/json -Uri $stageurl3 -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}
    $queueddeployments = $stageinfo3.value
    #write-output $queueddeployments
        if($inprogressdeployments) {            
            Write-output "Deployment In Progress, Waiting for it to finish!"  
            Write-output "Next attempt in 30 seconds"
            Start-sleep -Seconds 30          
      } else {            
            Write-Host "No Current Deployment in Progress"
            if($queueddeployments) {
            write-output "Current Queued deployments"
            Write-output "if 2 - Next attempt in 30 seconds"
            Start-sleep -Seconds 30 
            }
            else{
            write-output "No Queued deployments, starting release"
            $success = $true      
            }      
      }
    }
    catch{
        Write-output "catch - Next attempt in 30 seconds"
        write-output "1"
        Start-sleep -Seconds 30
      # Put the start-sleep in the catch statemtnt so we
      # don't sleep if the condition is true and waste time
    }
    
    $count++
    
}until($count -eq 2000 -or $success)
if(-not($success)){exit}

Result:

Stage1 will continue to check until all previous versions are complete.

enter image description here

We could also try it with demands, we could specify same demands. Use demands to make sure that the capabilities your pipeline needs are present on the agents that run it. It won't run unless one or more demands are met by the agent.

Vito Liu
  • 7,525
  • 1
  • 8
  • 17
  • I think this suffers from the same limitation as trigger: batch, i.e. it disallows concurrency at the pipeline level, not at the stage level. However, I think I can adapt this solution to be at the stage level by putting something very similar to this code as the first task in each UI Test stage, and additionally ensuring no other build is running the same stage. – Lee Richardson Dec 23 '20 at 18:34
  • If you are using self-hosted agent, you could try it with demands. And I believe that add the script to check previous release version is the best workaround. – Vito Liu Dec 24 '20 at 09:57
1

Your
UI Test environment 1 stage (specifically deployment job which Runs the tests) should target
env1uitest environment.

UI Test environment 2 stage (specifically deployment job which Runs the tests) should target
env2uitest environment.

Then set a exclusive lock ( https://learn.microsoft.com/en-us/azure/devops/release-notes/2020/pipelines/sprint-172-update#exclusive-deployment-lock-policy) policy on these two environments.
It should address your need: "ensuring that no two agents will ever perform a UI Test for a given environment at the same time"

https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals?view=azure-devops&tabs=check-pass#exclusive-lock

fenrir
  • 246
  • 1
  • 9
-1

You put the tasks into different jobs, and specify a dependsOn property for jobs that need to wait for particular job to be finished. If you leave out the dependsOn, it will run synchronounsly.

After reading your prompt more closely, I think you just need a build stage that builds and publishes your artifact.

then a deploy_test stage for each environment. Stages run sequentially by default (unlike jobs)

michiel Thai
  • 547
  • 5
  • 16
  • Sorry I probably asked poorly, or am not understanding this answer. It already does run sequentially for a single run, exactly how I want. The issue is when there are multiple simultaneous runs, e.g. if 2 people merge code at the exact same time, thus triggering 2 runs, and each run is handled by a different agent such that the runs happen in parallel. – Lee Richardson Dec 22 '20 at 17:27
  • In that case, trigger batch is your best bet. – michiel Thai Dec 23 '20 at 17:30