How to loop through user-defined variables in a YAML pipeline?

Question

I am trying to loop through user-defined variables in an Azure DevOps YAML pipeline.
The variables have been created through the UI:

Below the YAML pipeline code that I'm using:

trigger:
- dev
- main

pr:
- dev

pool:
  vmImage: ubuntu-latest

stages:

- stage:
  jobs:
  - job: TestVars
    steps:
    - ${{ each var in variables }}:
      - script: |
          echo ${{ var.key }}
          echo ${{ var.value }}
        displayName: ${{ var.key }}

When running the above pipeline only system and build variables are listed (e.g. system, system.hostType, build.queuedBy, etc.).

Any help to loop through user-defined variables would be much appreciated.

Have you tried mapping it like in the documentation?variables: GLOBAL_MY_MAPPED_ENV_VAR: $(nonSecretVariable) — Marko Marinkovic, Oct 28 '21 at 16:33
I did not try this, but my goal is to avoid manually mapping all variables. I would like to loop through them. I have no issue accessing variables individually with the `$(var)` syntax. — Chris, Oct 29 '21 at 06:27
I'm aware of variable groups, but since I don't need to share those variables between pipelines I was hoping there was a simpler solution. — Chris, Oct 30 '21 at 17:09
Do you have to use variables? If you could use [parameters](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/runtime-parameters?view=azure-devops&tabs=script#loop-through-parameters) instead, then looping through them is easy. — Vince Bowdren, Nov 06 '21 at 13:00

YK1 · Answer 1 · 2021-11-02T14:36:06.693

5

Unfortunately, no luck fetching the variables defined in UI. However, if your variables are non-secrets, you can bring them over into the YAML, and they will show up in the loop.

- stage:
  variables:
    myyamlvar: 1000   # this will show up in the loop
  jobs:
  - job: TestVars
    steps:
    - ${{ each var in variables }}:
      - script: |
          echo ${{ var.key }}
          echo ${{ var.value }}
        displayName: ${{ var.key }}

Alternatively, instead of using a compile time expression, you can list variables using a runtime construct, for example:

 - job: TestRuntimeVars
    steps:
      - script: |
          for var in $(compgen -e); do
            echo $var ${!var};
          done

This will list all variables including ones defined in the UI.

edited Nov 02 '21 at 14:36

answered Nov 02 '21 at 03:49

YK1

7,327
1
21
28

Hi, thanks for your answer, here you are showing a variable defined in the YAML, my question applies to variables defined in the UI at the pipeline level. – Chris Nov 02 '21 at 07:57
2

Since template expressions `${{ }}` are evaluated at compile time, I think variables defined in UI (including variable groups) may not be available, so defining variables in YAML is work around, if you need that loop. – YK1 Nov 02 '21 at 09:10
I understand, but in that case it's pretty much equivalent to mapping those variables to environment variables. I'm trying to avoid individual variable mapping in the YAML in favor of looping through them directly if that's possible. – Chris Nov 02 '21 at 13:48
1

not really, you don't need to map variables to environment if they are not secrets. I've updated answer with an example. – YK1 Nov 02 '21 at 14:23
I see your point, but I would like to have a mix of secret on non-secret variables defined through the UI. That's the purpose of my question, if it's not possible then I guess I will go for variable groups and/or key vault. – Chris Nov 02 '21 at 15:27
1

OK, Sure, but please note that variable groups are also not accessible via compile time template expressions and you will have to use a runtime construct to loop through. – YK1 Nov 02 '21 at 15:33

nicksnb · Answer 2 · 2021-11-03T02:57:10.700

From the Microsoft docs link you provided, it specifies that:

"Unlike a normal variable, they are not automatically decrypted into environment variables for scripts. You need to explicitly map secret variables."

However, one workaround could potentially be to run an azure cli task and get the pipeline variables using az pipelines variable list

Assuming your intention is to get the actual values, in which case maybe that won't suffice. Having said that, you should consider a variable group even if you're not using them in other pipelines since the group can be linked to an Azure KeyVault and map the secrets as variables. You can store your sensitive values in a KeyVault and link it to the variable group which can be used like regular variables in your pipeline.

Or you can access KeyVault secrets right from the AzureKeyVault pipeline task.

Thanks for your answer, however one thing is yet not clear to me; even user-defined non-secret variables are not displayed by my loop, do you know if they can be accessed? — Chris, Nov 02 '21 at 07:56
Also reading the docs, I don't expect the secret variables to be automatically mapped to environment variables, I just need to loop through them. — Chris, Nov 02 '21 at 09:43

milo.farrell · Answer 3 · 2021-11-06T12:57:53.640

To expand on the awnser below. It is a bit round about but you can use the azure devopps CLI. This may be a bit overkill but it does do the job.

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- bash: az --version
  displayName: 'Show Azure CLI version'

- bash: az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project=$(System.TeamProject) --use-git-aliases true
  displayName: 'Set default Azure DevOps organization and project'

- bash: |
    az pipelines variable list --pipeline-id $(System.DefinitionId)
  displayName: 'Show build list varibales'
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

This approach was taken from a combination of: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#list-variables and https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#list-variables

If the agent is self hosted you may need to install the dev opps cli.

How to loop through user-defined variables in a YAML pipeline?

3 Answers3

Linked