Azure DevOps yaml pipeline expression not evaluating

Question

I'm trying to assign the PublishTestResults@2 task's failTaskOnFailedTests argument in an azure-pipelines.yml file to be the result of an expression. The first time the tests are run, if there are any failed tests I want to fail the job. On subsequent attempts to Retry failed jobs I don't want to fail the job.

I've set up the task like this (second to last line is where I'm setting the failTaskOnFailedTests argument):

    - task: PublishTestResults@2
      displayName: 'Publish Test Results'
      inputs:
        testResultsFormat: 'JUnit'
        testResultsFiles: '*.xml'
        searchFolder: '$(testsOutputPath)'
        mergeTestResults: true
        failTaskOnFailedTests: eq($(System.JobAttempt), 1)
        testRunTitle: 'Test Results $(System.JobAttempt)'

When I run the pipeline with system diagnostics enabled I can see this in the log:

##[debug]testRunTitle=Test Results 1
##[debug]publishRunAttachments=true
##[debug]failTaskOnFailedTests=eq(1, 1)
##[debug]searchFolder=/home/vsts/work/1/s/TestProject/cypress/reports/junit
##[debug]testRunner: JUnit
##[debug]testResultsFiles: *.xml
##[debug]mergeResults: true

The third line shows failTaskOnFailedTests being set to the expression statement, not the evaluated value of the expression. I'm at a loss for what I'm doing wrong. The expression seems in line with others in the Expressions documentation.

What am I missing?

beatcracker · Accepted Answer · 2021-04-08T10:12:07.947

In Azure DevOps you can use compile time (${{ <expressions> }}) and runtime ($[ <expression> ]) expressions.

Compile time expressions are processed once, when template is rendered by the server. Runtime expressions are evaluated when pipeline executes.

In your case you'd need to use runtime expression, but they can only be used in variable assignments or condition parameter.

Example:

condition: eq(variables['System.JobAttempt'], '1')

So using this approach you can't configure failTaskOnFailedTests input directly, but you can skip the task entirely.

I've not tested this, but to achieve the desired result you could try to use a script to set failTaskOnFailedTests depending on the System.JobAttempt value.

Example:

- bash: echo "##vso[task.setvariable variable=FailTask]$((( $(System.JobAttempt) > 1 )) && echo false || echo true)"

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '*.xml'
    searchFolder: '$(testsOutputPath)'
    mergeTestResults: true
    failTaskOnFailedTests: $(FailTask)
    testRunTitle: 'Test Results $(System.JobAttempt)'

There is also a chance that manipulating variable directly might work.

Example:

variables:
  FailTask: $[ le( variables['System.JobAttempt'], 1 ) ]

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '*.xml'
    searchFolder: '$(testsOutputPath)'
    mergeTestResults: true
    failTaskOnFailedTests: $(FailTask)
    testRunTitle: 'Test Results $(System.JobAttempt)'

Setting the variable via a script, and using the variable, fixed it for me. Thank you for the assist! — BigPigVT, Apr 07 '21 at 20:28
@BigPigVT Good to hear, I've also added an example that uses variable assignment directly, so you might not need the script at all. — beatcracker, Apr 08 '21 at 10:13

Azure DevOps yaml pipeline expression not evaluating

1 Answers1