0

I have a single stage with 3 jobs:
Job A -> Always publishes artifact
Job B -> Always publishes artifact
Job C -> Depends on and uses the artifacts from Job A and Job B

Jobs A and B are retryable if they fail.

Ideally, jobs A and B would overwrite the artifact produced when retried but Yaml pipelines don't allow deleting artifacts and the suggested workaround is to put the Job ID in the artifact name.

How do I get the correct artifacts from Jobs A and B in Job C in the following scenario:
Pipeline is executed and Job A succeeds but Job B fails - both still publish artifacts which results in:
JobA_Artifact1
JobB_Artifact1

The failed Job B is retried and succeeds but then we end up with the following artifacts:
JobA_Artifact1
JobB_Artifact1
JobB_Artifact2

In Job C - how do I now get the latest artifact from Job A (which was from the previous execution) and the latest artifact from Job B? I'm hoping there's a better way than sticking a counter onto the end of the artifacts and then basically searching backwards for each artifact.

kmxp
  • 25
  • 6

1 Answers1

1

How to download artifacts from the last job execution when retrying a job in the same stage using Yaml pipelines?

Update:

After discussing with kmxp, I found that the solution I initially provided was not suitable for his situation. Let me share the final workaround.

Since the artifacts are not locally on the build agent but are published to the build pipeline artifact, so we could not clean it manually with some scripts.

To resolve this, we could use the variables for the artifact name, like:

JobA and JobB:

- powershell: |
    echo "##vso[task.setvariable variable=artifactJobId;isOutput=true]$(System.JobId)"
  Name: setArtifactJobId

JobC:

variables: 
  JobAArtifactId: $[ dependencies.JobA.outputs['setArtifactJobId.artifactJobId'] ]

But there is another situation that the variable is not accessible from Job C when the JobA or Jobb fail. We could add condition for above powershell task succeededOrFailed():

- powershell: |
    echo "##vso[task.setvariable variable=artifactJobId;isOutput=true]$(System.JobId)"
  Name: setArtifactJobId
  condition: succeededOrFailed()
Leo Liu
  • 71,098
  • 10
  • 114
  • 135
  • 1
    I think I wasn't clear enough in my description of the problem (I've updated the description in case that helps) - the artifacts I am talking about are not locally on the build agent but are published to the build pipeline so I don't think they are deletable. I'm currently trying appending the JobId to the artifact name and then creating a variable from the job (A and B) that's then used by the next job (Job C) through dependencies (JobAId: $[ dependencies.JobA.outputs['exportJobId.JobId'] ]) but it seems like maybe you cannot access dependency outputs if the job fails? I'm still testing it. – kmxp Sep 01 '21 at 08:03
  • @kmxp, Ok, I got it. You are right. Which task you are using to download the artifact? Is `Download Pipeline Artifact`? If yes, I think use the variables for the artifact name. – Leo Liu Sep 01 '21 at 08:27
  • Yes, that's the task I'm using for the download. I'm using a Powershell task to set the JobId variable from Jobs A and B but it definitely seems like when the Job fails, then the variable is not accessible from Job C - I'm just getting an empty variable even though the Powershell task is running. `- powershell: | echo "##vso[task.setvariable variable=artifactJobId;isOutput=true]$(System.JobId)" name: setArtifactJobId` And then in Job C: `variables: JobAArtifactId: $[ dependencies.JobA.outputs['setArtifactJobId.artifactJobId'] ]` – kmxp Sep 01 '21 at 08:44
  • @kmxp, How about set the condition for the powershell task in the Jobs A and B, `succeededOrFailed()`? https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml – Leo Liu Sep 01 '21 at 08:50
  • You're right, I mistakenly thought the Powershell task was running but it was skipped because I didn't have the succeededOrFailed() set (I did check but I must have checked the wrong Job). The variable is coming through now. I won't accept your answer because it's not what I was asking for and I'll post my own answer tomorrow to help anyone else in future but you're welcome to post a new answer / update your answer before then with the actual solution and I'd be happy to accept that instead. Thank you for helping figure this out. – kmxp Sep 01 '21 at 10:47
  • @kmxp, No worries about the accept answer. I have updated the answer in case other people waste time on incorrect answers. This is welcome you can post your more detailed answers, – Leo Liu Sep 02 '21 at 02:44