Azure Build Pipeline - multiple repositories built with multipe jobs - how to reuse the same working directories each build?

Question

I have a main solution and N other solutions in different repositories. Each time main solution builds I have to build N others and create an artifact of main solution containing others. My current build pipeline looks like this.

• [Stage 1]

  • [Job 1] Build Main solution -> publish pipeline artifact (needed for other solutions) [agent folder 1]

  • [Job 2..N] Checkout Solution "n" -> build -> publish pipeline artifact [agent folders 2..N]

• [Stage 2] - After all jobs succeded
  • [Job 1] - Download main artifact -> download other artifacts -> put them in one artifact -> publish [Agent folder N + 1]

My problem is that next time this pipeline runs, my agent recycles the last folder and the others are there waisting a lot of memory. Is there a way to tell the pipeline to run each job in the same agent working directory? Or to reuse the first directory created (the one for building main solution)?

EDIT: Here is the code of my pipeline

resources:
  repositories:
  - repository: Repo.With.Templates
    type: git
    name: Repo.With.Templates
    ref: feature/templates

  - repository: Secondary.Repository.1
    type: git
    name: Secondary.Repository.1
    ref: f/refactor/cd

  - repository: Secondary.Repository.2
    type: git
    name: Secondary.Repository.2
    ref: f/refactor/cd

trigger:
  batch: true
  branches:
    include:
    - 'f/core/cd-updates'

pool: 'Example pool'

variables:
 solution: '**/*.sln'
 buildPlatform: 'Any CPU'
 buildConfiguration: 'Release'
 scriptSetBuildNumber: 'CI\SetBuildNumber.ps1'
 nugetConfigFile: '$(Build.SourcesDirectory)/NuGet.config'
 checkoutFolderRoot: '$(Build.SourcesDirectory)'

stages:
- stage: BuildingStage
  jobs:
  - job: Main_Solution_Build
    steps:
    - task: VSBuild@1
      displayName: 'Build'
      inputs:
        solution: '$(solution)'
        msbuildArgs: '
            /p:DefineConstants="TESTENV"
            /p:TreatWarningsAsErrors=true
            /p:DeployDefaultTarget=WebPublish
            /p:WebPublishMethod=FileSystem
            /p:DeleteExistingFiles=True
            /p:SkipInvalidConfigurations=false
            /p:VisualStudioVersion=11.0
            /p:publishUrl="$(Build.ArtifactStagingDirectory)\"
            '
        platform: '$(buildPlatform)'
        configuration: '$(buildConfiguration)'

    # Zip bin folder and move to artifact folder
    - task: ArchiveFiles@2
      name: 'MovingMainSolutionBinToArtifacts'
      inputs:
        rootFolderOrFile: '$(Build.SourcesDirectory)/MainSolution/bin'
        includeRootFolder: true
        archiveType: 'zip'
        archiveFile: '$(Build.ArtifactStagingDirectory)/Artifacts/MainSolution.zip'
        replaceExistingArchive: true

    - task: PublishPipelineArtifact@1
      name: PublishMainSolutionBinPipeArtifact
      inputs:
        targetPath: '$(Build.ArtifactStagingDirectory)\Artifacts\MainSolution.zip'
        artifact: 'MainSolutionBinDrop'
        publishLocation: 'pipeline'

  - job: SecondarySolution1
    dependsOn: Main_Solution_Build
    steps:
    - checkout: Secondary.Repository
    - template: build-service-template.yml@Repo.With.Templates
      parameters:
        serviceName: "SecondarySolution1"


## Stage for assembling MainSolution and all the services together
- stage: Assemble
  dependsOn: BuildingStage
  jobs:
  - job: AssembleArtifact
    steps:
    - checkout: none

    - task: DownloadPipelineArtifact@2
      name: "MainSolutionBinDrop"
      inputs:
        buildType: 'specific'
        project: 'a12e0163-b207-400d-ac93-fa47964d5010'
        definition: '2'
        buildVersionToDownload: 'latest'
        artifactName: 'MainSolutionBinDrop'
        targetPath: '$(Build.ArtifactStagingDirectory)/MainSolutionBinDrop'

    - task: DownloadBuildArtifacts@0
      name: "DownloadServicesDrop"
      inputs:
        buildType: 'specific'
        project: 'a12e0163-b207-400d-ac93-fa47964d5010'
        pipeline: '2'
        buildVersionToDownload: 'latest'
        downloadType: 'single'
        artifactName: 'ServicesDrop'
        downloadPath: '$(Build.ArtifactStagingDirectory)'

    # Tasks that produce one artifact out of Main artifact and all Secondary solution artifacts
    # .....
    # .....

      # Publish
    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: '$(Build.ArtifactStagingDirectory)/MainWithSecondary.zip'
        artifact: 'MainWithSecondary'
        publishLocation: 'pipeline'

And the code of the template used to build N secondary solutions (in this example only one)

parameters:
- name: solution
  type: string
  default: '**/*.sln'

- name: buildPlatform
  type: string
  default: 'Any CPU'

- name: buildConfiguration
  type: string
  default: 'Release'

- name: nugetConfigFile
  type: string 
  default: '$(Build.SourcesDirectory)/NuGet.config'

- name: serviceName
  type: string

steps:

# Tasks that download main build artifact, build secondary solution and publishes secondary artifact
#
#

#Download main solution artifact
- task: DownloadPipelineArtifact@2
  inputs:
    buildType: 'current'
    artifactName: 'MainSolutionDrop'
    targetPath: '$(Agent.BuildDirectory)/MainSolution'

- task: VSBuild@1
  displayName: 'Build'
  inputs:
    solution: '$(solution)'
    msbuildArgs: '
    /p:TreatWarningsAsErrors=true 
    /p:DeployOnBuild=true
    /p:SkipInvalidConfigurations=false'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

# Create artifact folder
- task: PowerShell@2
  name: "CreateArtifactFolder"
  inputs:
    targetType: 'inline'
    script: |
      if (!(Test-Path "$(Build.ArtifactStagingDirectory)\${{ parameters.serviceName }}" -PathType Container)) {    
                New-Item -ItemType Directory -Path "$(Build.ArtifactStagingDirectory)\${{ parameters.serviceName }}"
      }


- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'ServicesDrop'
    publishLocation: 'Container'

Answer 1

Each job has its own default working directory and it is separate with other jobs, it is set the agent automatically and cannot be changed. So you cannot run each job in the same agent working directory.

There is a workaround to download the multiple repos under the same working folder and build them in a single job.

You can run git command in a powershell task to clone the multiple repos in the same working directory. And then set the build tasks point to each repo folder to build each solution. Check below example:

You will need to use PAT to anthenticate. Check here to generate PAT with Code read and write scope

- powershell: |
    git clone https://{PAT}@dev.azure.com/{org}/{proj}/_git/Repo1  #repo1 will be cloned into folder Repo1 under $(Build.SourcesDirectory)
    #cd Repo1 #cd the code folder

- powershell: |
    git clone https://{PAT}@dev.azure.com/{org}/{proj}/_git/Repo2  #repo1 will be cloned into folder Repo2 under $(Build.SourcesDirectory)
    #cd Repo2 #cd the code folder
  ....

- task: Build tasks1 #point the solution folder to $(Build.SourcesDirectory)/Repo1
    ...
- task: Build tasks2 #point the solution folder to $(Build.SourcesDirectory)/Repo2
    ...

#- task: Copy file task # copy the built artifacts to a specified folder.

- task: publish build artifacts task  #to publish repo1 artifacts
  ....
- task: publish build artifacts task  #to publish repo2 artifacts
  ....

You can also use Copy file task to move the built artifacts to different folders.

By using git clone command in the powershell task to clone the repos, you can combine your jobs and stages into just one job.

Update:

After checked above yaml pipeline and tested, I found the - checkout: Secondary.Repository caused new folders being created where rerun the pipeline.

The workaround is to use a powershell task to clone Secondary.Repository instead of using - checkout: Secondary.Repository. Please check out above workaround.

You can also report this issue to Microsoft by Reporting a problem(Click Report a problem and choose Azure devops)

Hope above helps.