Can I call an Azure Pipelines task once for each file in a folder?

Question

I'm trying to create a pipeline to run swagger-codegen for each Swagger document in a repo. I'd like to avoid having to keep the pipeline yaml updated for each API specification we generate. For most stages of this process, it's enough to loop through them in Powershell. But for publishing nuget and npm packages, it would be a lot easier to call another template (or otherwise repeat a task) for each one. (Publishing to our Azure feed via powershell in a pipeline escapes me.)

I've had no trouble passing in an array when calling another template manually. I can generate a JSON string of an array of the file names. But, when I try to use this as a variable, or as a runtime expression, I get a compile error and the pipeline doesn't even run:

azure-pipelines.yml:
...
- task: PowerShell@2
  name: getSpecFiles
  inputs:
    targetType: 'inline'
    script: |
      $SpecFiles = ConvertTo-Json -InputObject ([array] (get-item .\OpenApi\*.json | select -ExpandProperty Name)) -Compress # comes out to ["NotesAndTasks-1.json"]
      $setSpecFileNames = '##vso[task.setvariable variable=specFileNames]' + $SpecFiles; # I've also tried specifying it's an output variable with no luck
      Write-Output $SpecFiles
      Write-Output $setSpecFileNames
  displayName: 'Get list of spec files'
- template: generate-many-apis.yml
  parameters:
#    specFileNames: $(specFileNames)
#    specFileNames: $[specFileNames]
     specFileNames: ["NotesAndTasks-1.json"]


generate-many-apis.yml:
parameters:
- name: specFileNames
  type: object
  default: { }

steps:
- ${{ each value in parameters.specFileNames }}:
  - task: PowerShell@2
    displayName: Generate APIs 
    inputs:
      targetType: 'inline'
      script: |
        Write-Output "${{ value }}" # not currently calling the generate-api task; just testing

What would it take to run the generate-api task once for each Swagger .json? Or am I better off figuring out how to publish my npm and nuget packages from a script task?

Also, I'm able to change the parameter of generate-many-apis to a string, and successfully pass it that way -- but then the ${{ each }} loop fails.

score 0 · Answer 1 · answered Jun 28 '21 at 09:45

It cannot work

- ${{ each value in parameters.specFileNames }}:

as there is no way at the moment to set programatically a complex type. For AD it is a string and just it. If your script is just a powerhsell or similar you can do it purely in powershell. I mean call CovertFrom-Json and then run your script in the loop. But the whole needs to be put in the powershell not in the YAML.

Please take a look [here](https://developercommunity.visualstudio.com/t/variables-in-yaml-pipeline-are-not-allowing-to-def/812728#TPIN-N914394\):

The intent of yaml runtime parameters to be a string-to-string map. There are currently no plans to change the behaviors of runtime variables for backwards compatibility reasons. However, parameters are generally the tool to be used where possible. Configuring pipelines based on compile-time constants produces easer to understand workflows, and users can “Download Full Yaml” to see the final resolved templates without queueing a run.

I don't think this comment is accurate. I may have mistaken the question and if so will remove the associated posts. YAML supports the use of nested objects and as such the ADO implementation follows along. — DreadedFrost, Jun 28 '21 at 14:01

score 0 · Answer 2 · answered Jun 28 '21 at 14:00

It helps if tackling this a YAML question more than ADO question. This is possible as YAML supports the use of objects.

Here's and example of how to set this up. It's slightly different but the problem statement I believe is the same.

- name: environmentObjects
  type: object
  default: 
  - environmentName: 'dev'
    regionAbrvs: ['eus']
  - environmentName: 'uat'
    regionAbrvs: ['eus', 'cus']

Then passing the object to a template like:

  - ${{ each environmentObject in parameters.environmentObjects }} :
    - ${{ each deploymentObject in parameters.deploymentObjects }} :

        - template: stages/webapp_deploy_env_stage.yml@YAMLTemplates
          parameters:
            webAppName: ${{ deploymentObject.webAppName }}
            projectName: ${{ deploymentObject.projectName }}
            dependsOn: ${{ deploymentObject.dependsOn}}
            environmentName: ${{ environmentObject.environmentName}}
            regionAbrvs: ${{ environmentObject.regionAbrvs }}
            solutionName: ${{ parameters.solutionName}}

Notice that in this specific example regionAbrvs is an array inside the object and being passed as an object to the template.

Then inside that emplate another loop to go through the object being passed:

 jobs:
      - ${{ each regionAbrv in parameters.regionAbrvs }} :
        - template: ../jobs/webapp_deploy_job.yml@YAMLTemplates
          parameters:
            environmentName: ${{ parameters.environmentName}}
            webAppName: 'wapp-${{ parameters.webAppName}}-${{ parameters.environmentName}}-${{regionAbrv}}'
            regionAbrv: ${{ regionAbrv }}
            projectName: ${{ parameters.webAppName}}
            takeAppOfflineFlag: ${{ parameters.takeAppOfflineFlag}}
            packagePath: '$(Pipeline.Workspace)/drop/${{ parameters.projectName }}.zip'

Essentially if you have an object and define the loop like: - ${{ each valueString in parameters.specFileNames }}: Then in the loop reference each individual value as valueString

This is an example for Python but the concepts still apply as these are types defined in YAML.

Thank you but I'm able to provide objects or arrays as inputs to an Azure template, but the specific thing I'm trying to do is list each file in a folder as a parameter without duplicating the list in the YAML; I don't think this is covered in your example. — Peter Bailey, Jun 28 '21 at 14:37

Can I call an Azure Pipelines task once for each file in a folder?

2 Answers2