Download artifact from Azure DevOps Pipeline grandparent Pipeline

Question

Given 3 Azure DevOps Pipelines (more may exist), as follows:

1. Build, Unit Test, Publish Artifacts
2. Deploy Staging, Integration Test
3. Deploy Production, Smoke Test

How can I ensure Pipeline 3 downloads the specific artifacts published in Pipeline 1?

The challenge as I see it is that the Task DownloadPipelineArtifact@2 only offers a means to do this if the artifact came from the immediately preceding pipeline. By using the following Pipeline task:

- task: DownloadPipelineArtifact@2
  inputs:
    buildType: 'specific'
    project: '$(System.TeamProjectId)'
    definition: 1
    specificBuildWithTriggering: true
    buildVersionToDownload: 'latest'
    artifactName: 'example.zip'

This works fine for a parent "triggering pipeline", but not a grandparent. Instead it returns the error message:

Artifact example.zip was not found for build nnn.

where nnn is the run ID of the immediate predecessor, as though I had specified pipelineId: $(Build.TriggeredBy.BuildId). Effectively, Pipeline 3 attempts to retrieve the Pipeline 1 artifact from Pipeline 2. It would be nice if that definition: 1 line did something, but alas, it seems to do nothing when specificBuildWithTriggering: true is set.

Note that buildType: 'latest' isn't safe; it appears it permits publishing an untested artifact, if emitted from Pipeline 1 while Pipeline 2 is running.

There may be no way to accomplish this with the DownloadPipelineArtifact@2. It's hard to be sure because the documentation doesn't have much detail. Perhaps there's another reasonable way to accomplish this... I suppose publishing another copy of the artifact at each of the intervening pipelines, even the ones that don't use it, is one way, but not very reasonable. We could eliminate the ugly aspect of creating copies of the binaries, by instead publishing an artifact with the BuildId recorded in it, but we'd still have to retrieve it and republish it from every pipeline.

If there is a way to identify the original CI trigger, e.g. find the hash of the initiating GIT commit, I could use that to name and refer to the artifacts. Does Build.SourceVersion remain constant between triggered builds? Any other "Initiating ID" would work equally well.

You are welcome to comment on the example pipeline scenario, as I'm actually currently using it, but it isn't the point of my question. I think this problem is broadly applicable, as it will apply when building dependent packages, or for any other reasons for which "Triggers" are useful.

Answer 1

An MS representative suggested using REST for this. For example:

HTTP GET https://dev.azure.com/ORGNAME/PROJECTGUID/_apis/build/Builds/2536

-

{
    "id": 2536,
    "definition": {
        "id": 17
    },
    "triggeredByBuild": {
        "id": 2535,
        "definition": {
            "id": 10
        }
    }
}

By walking the parents, one could find the ancestor with the desired definition ID (e.g. 10). Then its run ID (e.g. 2535) could be used to download the artifact.

@merlin-liang-msft suggested a similar process for a different requirement from @sschmeck, and their answer has accompanying code.

Answer 2

There are extensions that allow you to do this, but the official solution it to use a multi-stage pipeline and not 3 independent pipelines.

Answer 3

One way is using release pipelines (you can't code/edit it in YAML) but you can use the same artifacts through whole deployment.

Release pipeline

You can also specify required triggers to start deployment on

Approval and triggers

Alternatively, there exist multi-stage pipeline, that are in preview.(https://devblogs.microsoft.com/devops/whats-new-with-azure-pipelines/ ). You can access it by enabling it in your "preview feature".

Answer 4

Why don't you output some pipeline artifacts with meta info and concatenate these in the preceding pipes like.

Grandparent >meta about pipe Parent > meta about pipe and grantparent meta

Etc