Best practice for maintaining build order in Azure DevOps?

Question

What is the best way to maintain a certain build order/sequence in Azure DevOps? I cannot find any documentation discussing this.

Example: I have Pipeline A, B, C, and D. Note that each pipeline can be in different team projects.

Pipeline A and B must be ran before Pipeline C, since Pipeline C consumes output from A and B. A and B CAN be built in parallel but both must be complete before C runs. Pipeline D needs the output from Pipeline C, so Pipeline D can only begin once C is done.

I am thinking of using Stages to accomplish this.. for example Stage 1: Call Azure DevOps REST API to kick off Pipelines A and B and wait for their completion. Stage 2: Calls the API to build pipeline C Stage 3: Call rest API for pipeline D or just run the steps for pipeline D here (if it's the last in the build order).

Is there a better way to do this? I am wondering if pipeline resources can accomplish this but I don't think they maintain build order...I need outputs from previous pipelines but their run times can vary.

Answer 1

If you need such orchestration you have to rely on your own solution. Pipeline resource won't help because if you define multiple in one pipeline they will fire on completion any of defined pipeline.

However what you should consider is:

• If pipeline A runs twice, how does it impact on master plan
• Can pipeline A be run separately? If yes, what if it happens during execution of master plan? (question is similar to the one above)

If you want to implement this you can use another pipeline to try to orchestrate this, or Azure Function and webhooks, so Azure Function will be notified about each finish of the pipelines ABCD and once all is finished can fire sth else.

I think that this would be easier then actively waiting for the finish.

Answer 2

If A, B, C and D are four individual pipelines. You can try like as below.

1. In pipeline A:

• Add a job (suppose named job_top) at the top to execute the REST API "Queues a build" to trigger pipeline B. And from the Response, you can get the triggered buildId of pipeline B.

• Add a job (suppose named job_bottom) at the bottom to execute the REST API "Gets a build" to check if the triggered build of pipeline B has completed. The buildId is passed from above job. You can execute this API every few seconds or minutes in a loop until the pipeline B is completed.

• If pipeline C and A are not in the same project, add a step in job_bottom to execute the REST API "Queues a build" to trigger pipeline C, after the REST API "Gets a build" confirms that pipeline B has completed.

• If pipeline C and A are in the same project, you can use the build completion triggers to trigger pipeline C after pipeline A is completed. More details, see "Trigger one pipeline after another".

2. In pipeline C, to trigger pipeline D after pipeline C is completed, similar to above:

• If pipeline C and D are not in the same project, use the REST API "Queues a build" to trigger pipeline D.

• If pipeline C and D are in the same project, use the build completion triggers to trigger pipeline D after pipeline C is completed.

With above workaround, you just need to trigger pipeline A, then pipeline B, C and D will be automatically trigger in order.

Answer 3

You could add a PowerShell task after the Trigger Pipeline Task (This task is used to trigger the pipeline).

Pipeline sample:

 steps:
  - task: TriggerPipeline@1 #trigger Pipeline A
  - task: TriggerPipeline@1 #trigger Pipeline B
  - task: PowerShell@2      #check Pipeline A and B Status
  - task: TriggerPipeline@1 #trigger Pipeline C
  - task: PowerShell@2      #check Pipeline C Status
  - task: TriggerPipeline@1 #trigger Pipeline D

Here is Powershell script sample(Add a Powershell task after the task A and B to check two pipelines status at the same time):

$token = "PAT"

$url="https://dev.azure.com/{OrganizationName}/{ProjectName}/_apis/build/definitions/{DefinitionID}?includeLatestBuilds=true&api-version=5.1"

$url1="https://dev.azure.com/{OrganizationName}/{ProjectName}/_apis/build/definitions/{DefinitionID}?includeLatestBuilds=true&api-version=5.1"

$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))



$response = Invoke-RestMethod -Uri $url -Headers @{Authorization = "Basic $token"} -Method Get -ContentType application/json


$response1 = Invoke-RestMethod -Uri $url1 -Headers @{Authorization = "Basic $token"} -Method Get -ContentType application/json


$buildid = $response.latestBuild.id
echo $buildid

$buildid1 = $response1.latestBuild.id
echo $buildid1

$success = $false

do{
    try{
    $Buildurl2 = "https://dev.azure.com/{OrganizationName}/{ProjectName}/_apis/build/builds/$($buildid)?api-version=5.0"


    $Buildinfo2 = Invoke-RestMethod -Method Get -ContentType application/json -Uri $Buildurl2 -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}

    $BuildStatus= $Buildinfo2.status 
    $result = $Buildinfo2.result
    echo $result
    echo $BuildStatus

    $Buildurl3 = "https://dev.azure.com/{OrganizationName}/{ProjectName}/_apis/build/builds/$($buildid1)?api-version=5.0"
    $Buildinfo3 = Invoke-RestMethod -Method Get -ContentType application/json -Uri $Buildurl3 -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}
    $BuildStatus1= $Buildinfo3.status 
    $result1 = $Buildinfo3.result

    echo $result1
    echo $BuildStatus1
 
   
        if($BuildStatus -eq "completed" -and $BuildStatus1 -eq "completed" ) {            

            write-output "No Running Pipeline, starting Next Pipeline"
            $success = $true 
                       
      } else {   
            Write-output "Pipeline Build In Progress, Waiting for it to finish!"  
            Write-output "Next attempt in 30 seconds"
            Start-sleep -Seconds 30         

            }
    
      
    }
    catch{
        Write-output "catch - Next attempt in 30 seconds"
        write-output "1"
        Start-sleep -Seconds 30
      # Put the start-sleep in the catch statemtnt so we
      # don't sleep if the condition is true and waste time
    }
    
    $count++
    
}until($count -eq 2000 -or $success -eq $true )
if ($result -ne "succeeded" -or $result1  -ne "succeeded")
{
   echo "##vso[task.logissue type=error]Something went very wrong."
}

if(-not($success)){exit}

Explanation:

This powershell script runs the following two Rest APIs：

Definitions - Get

Builds - Get

For checking the status of a piepline, you can refer to my other ticket.