using for-loop in azure pipeline jobs

Question

I'm gonna use a for-loop which scans the files (value-f1.yaml, values-f2.yaml,...) in a folder and each time use a filename as a varibale and run the job in Azure pipeline job to deploy the helmchart based on that values file. The folder is located in the GitHub repository. So I'm thinking of something like this:

pipeline.yaml

 stages:
  - stage: Deploy
    variables:
      azureResourceGroup: ''
      kubernetesCluster: ''
      subdomain: ''
    jobs:
    ${{ each filename in /myfolder/*.yaml}}:
       valueFile: $filename 
       - template: Templates/deploy-helmchart.yaml@pipelinetemplates    

deploy-helmchart.yaml

 jobs:
    - job: Deploy
      pool:
        vmImage: 'ubuntu-latest'
      steps:
      - task: HelmInstaller@1
        displayName: 'Installing Helm'
        inputs:
          helmVersionToInstall: '2.15.1'
        condition: and(succeeded(), startsWith(variables['build.sourceBranch'], 'refs/tags/v'))  

      - task: HelmDeploy@0
        displayName: 'Initializing Helm'
        inputs:
          connectionType: 'Azure Resource Manager'
          azureSubscription: $(azureSubscription)
          azureResourceGroup: $(azureResourceGroup)
          kubernetesCluster: $(kubernetesCluster)
          command: 'init'
        condition: and(succeeded(), startsWith(variables['build.sourceBranch'], 'refs/tags/v'))  

      - task: PowerShell@2
        displayName: 'Fetching GitTag'
        inputs:
          targetType: 'inline'
          script: |
            # Write your PowerShell commands here.
            Write-Host "Fetching the latest GitTag"
            $gt = git describe --abbrev=0
            Write-Host "##vso[task.setvariable variable=gittag]$gt"
        condition: and(succeeded(), startsWith(variables['build.sourceBranch'], 'refs/tags/v'))    


      - task: Bash@3
        displayName: 'Fetching repo-tag'
        inputs:
          targetType: 'inline'
          script: |
            echo GitTag=$(gittag)
            echo BuildID=$(Build.BuildId)
            echo SourceBranchName=$(Build.SourceBranchName)
            echo ClusterName= $(kubernetesCluster)

      - task: HelmDeploy@0
        displayName: 'Upgrading helmchart'
        inputs:
          connectionType: 'Azure Resource Manager'
          azureSubscription: $(azureSubscription)
          azureResourceGroup: $(azureResourceGroup)
          kubernetesCluster: $(kubernetesCluster)
          command: 'upgrade'
          chartType: 'FilePath'
          chartPath: $(chartPath)
          install: true
          releaseName: $(releaseName)
          valueFile: $(valueFile)
          arguments: '--set image.tag=$(gittag) --set subdomain=$(subdomain)'
        condition: and(succeeded(), startsWith(variables['build.sourceBranch'], 'refs/tags/v'))  

Another thing is that if the jobs can get access to the GitHub repo by default or do I need to do something in the job level?

Besides how can I use for-loop in the job for this case?

Any help would be appreciated.

Updated after getting comments from @Leo

Here is a PowerShell task that I added in deploy-helmchart.yaml for fetching the files from a folder in GitHub.

 - task: PowerShell@2
   displayName: 'Fetching Files'
   inputs:
     targetType: 'inline'
     script: |
       Write-Host "Fetching values files"
       cd myfolder
       $a=git ls-files
       foreach ($i in $a) {
          Write-Host "##vso[task.setvariable variable=filename]$i"
          Write-Host "printing"$i
       }

Now the question is how can I run the task: HelmDeploy@0 for each files using parameters?

Answer 1

if the jobs can get access to the GitHub repo by default or do I need to do something in the job level?

The answer is yes.

We could add a command line task in the jobs, like job1 to clone the GitHub repository by Github PAT, then we could access those files (value-f1.yaml, values-f2.yaml,...) in $(Build.SourcesDirectory):

git clone https://<GithubPAT>@github.com/XXXXX/TestProject.git

Besides how can I use for-loop in the job for this case?

You could create a template which will have a set of actions, and pass parameters across during your build, like:

deploy-helmchart.yaml:

parameters:
  param : []

steps:
  - ${{each filename in parameters.param}}:
    - scripts: 'echo ${{ filename  }}'

pipeline.yaml:

steps:
 - template: deploy-helmchart.yaml
   parameters:
     param: ["filaname1","filaname2","filaname3"]

Check the document Solving the looping problem in Azure DevOps Pipelines for some more details.

Command line get the latest file name in the foler:

   FOR /F "delims=|" %%I IN ('DIR "$(Build.SourcesDirectory)\*.txt*" /B /O:D') DO SET NewestFile=%%I
    
   echo "##vso[task.setvariable variable=NewFileName]NewestFile"

Update:

Now the question is how can I run the task: HelmDeploy@0 for each files using parameters?

Its depends on whether your HelmDeploy` task has options to accept the filename parameter.

As I said before, we could use following yaml to invoke the template yaml with parameters:

 - template: deploy-helmchart.yaml
   parameters:
     param: ["filaname1","filaname2","filaname3"]

But, if the task HelmDeploy has no options to accept parameters, we could not run the task HelmDeploy@0 for each files using parameters.

Then I check the HelmDeploy@0, I found there is only one option that can accept Helm command parameters:

So, the answer for this question is depends on whether your file name can be used as a Helm command, if not, you could not run the task HelmDeploy@0 for each files using parameters. If yes, you can do it.

Please check the official document Templates for some more details.

Hope this helps.