Howto: Dynamic variable name resolution in Azure DevOps YAML

Question

The consistency of Variable support & the syntax vary wildly in Azure DevOps YAML. Case in point:

trigger:
- master

# Variable Group has $(testCategory1) with value
# 'TestCategory=bvttestonly | TestCategory=logintest'
variables:
  - group: DYNAMIC_VG

jobs:
  - job:
    pool: 'MyPool' #Has about 10+ self hosted agents

    strategy:
      parallel: $[ variables['noOfVMsDynamic']]

    variables:
      indyx: '$(testCategories$(System.JobPositionInPhase))'
      indyx2: $[ variables['indyx'] ] 
      testCategories: $[ variables[ 'indyx2' ] ]

    steps:
    - script: |
        echo "indyx2 - $(indyx2)"
        echo "testCategories $(testCategories)"
      displayName: 'Display Test Categories'

The step prints:

"indyx2 - $(testCategories1)"
"testCategories $(testCategories1)"

I need to print the value of $(testCategories1) defined in the Variable Group:

'TestCategory=bvttestonly | TestCategory=logintest'

have you tried `indyx: $[ variables[format('{0}{1}', 'testCategories', variables['System.JobPositionInPhase'])] ]` ? can't really test this right now, so posting as a comment, you may need to change that `.` to something else — Luizgrs, Jun 04 '20 at 18:38
Workaround: Use dynamic Matrix strategy with JSON created at run time. Refer to https://stackoverflow.com/questions/59076036/generate-job-matrix-from-all-possible-combinations-of-input-parameters/62649918#62649918 — Vyas Bharghava, Jun 30 '20 at 03:43

score 5 · Answer 1 · answered Jun 05 '20 at 13:02

5

This may work to you:

variables
   indyx: $[ variables[format('{0}{1}', 'testCategories', variables['System.JobPositionInPhase'])] ]

It worked for me, in a slightly different situation which also required some dynamic variable names.

answered Jun 05 '20 at 13:02

Luizgrs

4,765
1
22
28

score 2 · Answer 2 · answered Mar 18 '20 at 09:50

2

Howto: Dynamically resolve a nested variable in Azure DevOps YAML

That because the value of nested variables (like $(testCategories$(System.JobPositionInPhase))) are not yet supported in the build pipelines at this moment.

That the reason why you always get the value $(testCategories1) rather than the real value of variable testCategories1.

I encountered this issue many times in my past posts and we do not have a perfect solution before Azure Devops supports this feature.

For the convenience of testing, I simplified your yaml like following:

jobs:
  - job: ExecCRJob
    timeoutInMinutes: 800

    pool:
      name: MyPrivateAgent

    displayName: 'Execute CR'


    variables:
      testCategories1: 123456
      testCategoriesSubscripted: $(testCategories$(System.JobPositionInPhase))

    strategy:
      parallel: $[variables['noOfVMs']]     
    steps:
    - template: execute-cr.yml
      parameters:
        testCategories: $(testCategoriesSubscripted)

The execute-cr.yml:

steps:
    - script: echo ${{ parameters.testCategories }}

We always get the $(testCategories1)NOT the value of it.

If I change the $(testCategories$(System.JobPositionInPhase)) to $(testCategories1), everything work fine.

Since nested variables are not yet supported, As workaround, we need to expand the nested variables for each value of testCategories, like:

- job: B
  condition: and(succeeded(), eq(dependencies.A.outputs['printvar.skipsubsequent'], 'Value1'))
  dependsOn: A
  steps:
  - script: echo hello from B

Check the Expressions Dependencies for some more details.

Hope this helps.

answered Mar 18 '20 at 09:50

Leo Liu

71,098
10
114
135

1

Thanks, Leo for confirming the nested variables do not work. I understand how dependencies object work. Now, how to the hard part: How do I simulate this with an alternative approach? Dynamically provide values to parallel jobs that use parallel or matrix strategies? I will only know how many agents I need at runtime through the 'noOfVMs' pipeline variable. – Vyas Bharghava Mar 18 '20 at 16:32
@VyasBharghava, hm, yes, you are right. Although we can split your tasks into separate jobs with dependencies to resolve the nested variables, we could not get the value of `$(System.JobPositionInPhase)` in one of job. This is a big challenge, I will try to check if there are other ways to solve this problem. – Leo Liu Mar 19 '20 at 09:57
Thank you!!! I did a lot of work in the last couple of days to see what works. So, it doesn't matter where the variables are (cross job, local or even in VGs), the variable won't resolve. I think this issue is ONLY in parallel. Without this, I'm not able to use parallel for anything useful. I want to distribute user supplied test categories and test concurrently OR revert all active agents in a different pool to a previous snapshot (which will upon restart will install & test a windows thick client) etc. – Vyas Bharghava Mar 21 '20 at 02:13
@VyasBharghava, Indeed, It seems that this problem is stuck in an endless loop. If we want to resolved the nested variables, we need split the tasks into separate jobs. But, in this case, `$(System.JobPositionInPhase)` variable will be invalid. – Leo Liu Mar 23 '20 at 09:26
I workaround using a matrix strategy where I hardcode all variable references (Say, biggest I expect is 20 parallel jobs), with each job in matrix directly referencing the variable: $(testCategory1)...to $(testCategory20). I plan to test this value in a shell script task and make it skip if the value is blank. The succeeding tasks get skipped and job ends. However, this would mean upto 20 jobs will scheduled regardless of whether only 2 jobs are needed (meaning testCategory3 onwards are blank). Could the product team provide an official workaround for this, please? – Vyas Bharghava Mar 23 '20 at 16:35
Also, I understand this can be unreasonable to expect more than one pass at variable dereferencing to arrive at a value: '$(var$(index))' . However, using the variable name as a look up key string in variables['testCategory${System.JobPositionInPhase)'] must work at least – Vyas Bharghava Mar 25 '20 at 20:50

score 1 · Answer 3 · answered Aug 31 '22 at 13:25

I manage to get dynamic name resolution by using get-item to read the corresponding environment variable, allowing construction of the name of the variable and then getting the value.

In our case we save the name of an autogenerated branch into a variable group and each repository will have its own variable.

$branchVarName = "$(Build.Repository.Name).BranchName".replace(".","_")
$branchName = (get-item -Path Env:$branchVarName).value
write-host "/$(System.TeamProject)/_apis/build/builds?&repositoryId=$(Build.Repository.ID)&repositoryType=TFSGit&branchName=refs/heads/$branchName&api-version=6.0"

Notice in the second line that I reference the variable content using .value because get-item returns a name/value key pair.

This extraction has to be done in script but could be exposed as an output variable if needed in another context.

score 0 · Answer 4 · answered Mar 18 '20 at 03:59

0

If I'm understanding your issue correctly, the problem is that the pipeline evaluates all variables at the runtime of the job. The solution in this scenario is to split your tasks into separate jobs with dependencies.

Have a look at my answer in this post and let me know if it's what you're after : YAML pipeline - Set variable and use in expression for template

answered Mar 18 '20 at 03:59

Bevan

1,305
1
11
17

I only need runtime evaluation. The parallel strategy automatically slices the steps mentioned below into separate jobs. I do have dependency already setup between jobs. I'm able to access the output variables of ExecPrerequisiteJob in ExecCRJob just fine. Only issue is the subscripted access. – Vyas Bharghava Mar 18 '20 at 16:35

Howto: Dynamic variable name resolution in Azure DevOps YAML

4 Answers4