Failed Test does not fail task in YAML

Question

I have a build pipeline as shown below

    variables:
    - name: BuildParameters.solution
      value: '**/*.sln'
    - name: buildConfiguration
      value: Release
    name: $(date:yyyyMMdd)$(rev:.r)
    trigger: none
    resources:
      repositories:
      - repository: self
        type: git
    stages:
    - stage: __default
      jobs:
      - job: Job_1
        displayName: Agent job 1
        pool:
          vmImage: windows-2019
        steps:
        - task: UseDotNet@2
          displayName: '.NET Core 3.1.404'
          inputs:
            version: '3.1.404'
            packageType: sdk
        - task: DotNetCoreCLI@2
          enabled: true
          displayName: dotnet test
          continueOnError: true
          timeoutInMinutes: 0
          inputs:
            command: test
            projects: '**/*test/*.csproj'
            arguments: --configuration $(buildConfiguration) --verbosity n --collect "Code coverage"
            publishTestResults: true        
        - task: PublishTestResults@2
          inputs:
            testResultsFormat: XUnit
            testResultsFiles: '**/TEST.xml'
            mergeTestResults: true
            failTaskOnFailedTests: true

I have broken a test on purpose

This test fails, but the publish test results task doesnt fail outright

I get the warning

##[warning]No test result files matching **/TEST.xml were found. xunit

Has anyone ever had this?

Annoyingly if a test fails the Azure pipeline classes this as a partial success which is not ideal. I am trying to get into a situation where if a test fails, my pipeline fails

This pipeline is used to verify pull requests - i.e. do a build of the code in the PR and ensure that all tests pass

When I searched for this there were references to installing .NET 4.6.2 or later, but this is being used inside a build pipeline, so not sure how to do that

Have I missed something in the dot net test task?

Paul

score 2 · Accepted Answer · answered Dec 24 '20 at 02:30

When you set publishTestResults field to true for dotnet test task. See below extract from here.

This option appends --logger trx --results-directory $(Agent.TempDirectory) to the command line arguments.

Above commands will generate the test result in VSTest(trx) format and output the result to folder $(Agent.TempDirectory)

So you need to change the PublishTestResults@2 task like below:

 - task: PublishTestResults@2
         inputs:
           testResultsFormat: VSTest  #change the format to VSTest
           testResultsFiles: '**/*.trx'  #change to find .trx file
           mergeTestResults: true
           failTaskOnFailedTests: true
           searchFolder: '$(Agent.TempDirectory)' #search the .trx file in $(Agent.TempDirectory)

Usually the continueOnError field is set to false by default. And the dotnet test task will fail if there is any test fails, See below:

If the continueOnError field is set to true. The dotnet test task and the pipeline will show Warnig state.

So you can have a try setting the continueOnError field to false, which will fail the task and pipeline if there is any test fails. And there will not be no need to use PublishTestResults task. For publishTestResults is set to true for dotnet test task, the test result will be automatically published by the dotnet test task. See below:

 - task: DotNetCoreCLI@2
          enabled: true
          displayName: dotnet test
          continueOnError: false
          timeoutInMinutes: 0
          inputs:
            command: test
            projects: '**/*test/*.csproj'
            arguments: --configuration $(buildConfiguration) --verbosity n --collect "Code coverage"
            publishTestResults: true

Thanks this looks hopeful, however, I have seen that there is an option for mergetestresults in the publish test results task but I don't see how to do this in the dotnet test task? — Paul, Dec 24 '20 at 11:28
@Paul dotnet test task donot have the mergetestresults option. If you meant showing the test results of different test projects in the Tests tab on the build summary page, using only dotnet test task can show all the test results in the Tests tab too. If you have to use the mergetestresults option. Then you can just change your publishtestresult task as above example. — Levi Lu-MSFT, Dec 25 '20 at 08:06

Failed Test does not fail task in YAML

1 Answers1