How to publish Cobertura coverage report from the VS coverage produced by dotnet test --collect "Code coverage"?

Question

So the process can be deduced from https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/dotnet-core?view=azure-devops#collect-code-coverage and contains of the following steps:

Run the tests with coverage without publishing the test results, because we need to control the location of the generated binary coverage report (see How to set a custom coverage result file path when running dotnet test --collect "Code coverage"?)
Convert the VS binary coverage result to XML
Install the reportgenerator tool
Convert the VS XML coverage to Cobertura using the reportgenerator tool
Publish the Cobertura report using the Publish Code Coverage Azure DevOps task
Publish the Test Results

These steps can be implemented with the following YAML template:

parameters:
  BuildConfiguration: Debug

steps:
- task: DotNetCoreCLI@2
  name: Test
  displayName: Test
  inputs:
    command: 'test'
    publishTestResults: false
    arguments: '-c ${{ parameters.BuildConfiguration }} --no-build -l trx -r $(Common.TestResultsDirectory)\tests --collect "Code coverage"'

- task: PublishTestResults@2
  displayName: Publish Test Results
  inputs:
    testResultsFormat: VSTest
    testResultsFiles: '*.trx'
    searchFolder: $(Common.TestResultsDirectory)\tests
    testRunTitle: $(Build.DefinitionName)-$(Build.BuildNumber)
  condition: succeededOrFailed()

- powershell: |
    $cc = "$(Get-ToolFolderFromNuGet Microsoft.CodeCoverage)\..\build\netstandard1.0\CodeCoverage\CodeCoverage.exe"
    $BinaryCoverageFile = (Get-Item "$(Common.TestResultsDirectory)\tests\*\*.coverage").FullName
    & $cc analyze /output:$(Common.TestResultsDirectory)\vstest-coverage.xml $BinaryCoverageFile
  displayName: Convert Coverage Result To Xml

- task: DotNetCoreCLI@2
  inputs:
    command: custom
    custom: tool
    arguments: install --tool-path . dotnet-reportgenerator-globaltool
  displayName: Install ReportGenerator tool

- script: .\reportgenerator.exe -reports:$(Common.TestResultsDirectory)\vstest-coverage.xml -targetdir:$(Common.TestResultsDirectory)\coverage\report -reporttypes:"Cobertura"
  displayName: Create Cobertura Coverage Report

- task: PublishCodeCoverageResults@1
  displayName: Publish Coverage Results
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: '$(Common.TestResultsDirectory)/coverage/report/Cobertura.xml'
    failIfCoverageEmpty: true

It uses my powershell function Get-ToolFolderFromNuGet to download a package from NuGet, but other than that no custom code is used.

Anyway, the problem is that the Publish Test Results task publishes the binary coverage result as a hyperlink in the same tab where the coverage results are supposed to be:

This is useless. If I comment out the Publish Test Results, the coverage page shows the expected result:

But now I lost the Tests page, of course:

Does anyone know how to resolve this conundrum?

P.S.

I opened a bug here - https://github.com/microsoft/azure-pipelines-tasks/issues/12670

EDIT 1

I tried to Publish the Test Results at the end and even deleting the binary coverage result file. Nothing helps.

score 1 · Accepted Answer · answered Apr 10 '20 at 17:22

By inspecting the diag logs for the Publish Test Results task I managed to figure out how to resolve it. Indeed, from the logs:

2020-04-10T15:48:12.1799774Z ##[debug]pattern: '*.trx'
2020-04-10T15:48:12.1839752Z ##[debug]findPath: 'd:\_wf\06\13\TestResults\tests'
2020-04-10T15:48:12.1840795Z ##[debug]statOnly: 'false'
2020-04-10T15:48:12.1843783Z ##[debug]findPath: 'd:\_wf\06\13\TestResults\tests'
2020-04-10T15:48:12.1844681Z ##[debug]findOptions.allowBrokenSymbolicLinks: 'true'
2020-04-10T15:48:12.1845119Z ##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
2020-04-10T15:48:12.1845244Z ##[debug]findOptions.followSymbolicLinks: 'true'
2020-04-10T15:48:12.1850467Z ##[debug]  d:\_wf\06\13\TestResults\tests (directory)
2020-04-10T15:48:12.1858807Z ##[debug]  d:\_wf\06\13\TestResults\tests\1f627391-dd45-48b3-af92-5213c471eb04 (directory)
2020-04-10T15:48:12.1864156Z ##[debug]  d:\_wf\06\13\TestResults\tests\tfsbuild_TDC5DFC1BLD24_2020-04-10_11_47_29 (directory)
2020-04-10T15:48:12.1868508Z ##[debug]  d:\_wf\06\13\TestResults\tests\tfsbuild_TDC5DFC1BLD24_2020-04-10_11_47_29\In (directory)
2020-04-10T15:48:12.1873069Z ##[debug]  d:\_wf\06\13\TestResults\tests\tfsbuild_TDC5DFC1BLD24_2020-04-10_11_47_29\In\TDC5DFC1BLD24 (directory)
2020-04-10T15:48:12.1880777Z ##[debug]  d:\_wf\06\13\TestResults\tests\tfsbuild_TDC5DFC1BLD24_2020-04-10_11_47_29\In\TDC5DFC1BLD24\tfsbuild_TDC5DFC1BLD24_2020-04-10.11_47_23.coverage (file)
2020-04-10T15:48:12.1884045Z ##[debug]  d:\_wf\06\13\TestResults\tests\tfsbuild_TDC5DFC1BLD24_2020-04-10_11_47_29.trx (file)
2020-04-10T15:48:12.1885576Z ##[debug]7 results

So the binary coverage file is found there. But its original location is different, we can see it from the regular build log of the test task itself:

2020-04-10T15:47:54.0912521Z M i c r o s o f t   ( R )   C o v e r a g e   C o l l e c t i o n   T o o l   V e r s i o n   1 6 . 0 . 3 0 3 1 9 . 3 0 0 2 
2020-04-10T15:47:54.0913009Z  
2020-04-10T15:47:54.0913107Z  
2020-04-10T15:47:54.0913209Z  C o p y r i g h t   ( c )   M i c r o s o f t   C o r p o r a t i o n .     A l l   r i g h t s   r e s e r v e d . 
2020-04-10T15:47:54.0913270Z  
2020-04-10T15:47:54.0913307Z  
2020-04-10T15:47:54.0913340Z  
2020-04-10T15:47:54.0913400Z  
2020-04-10T15:47:54.3182888Z  Results File: d:\_wf\06\13\TestResults\tests\tfsbuild_TDC5DFC1BLD24_2020-04-10_11_47_29.trx
2020-04-10T15:47:54.3183717Z 
2020-04-10T15:47:54.3184201Z Attachments:
2020-04-10T15:47:54.3184419Z   d:\_wf\06\13\TestResults\tests\1f627391-dd45-48b3-af92-5213c471eb04\tfsbuild_TDC5DFC1BLD24_2020-04-10.11_47_23.coverage
2020-04-10T15:47:54.3196805Z Test Run Successful.
2020-04-10T15:47:54.3197219Z Total tests: 445
2020-04-10T15:47:54.3197713Z      Passed: 445
2020-04-10T15:47:54.3199584Z  Total time: 30.1160 Seconds

So, the original binary coverage file is in d:\_wf\06\13\TestResults\tests\1f627391-dd45-48b3-af92-5213c471eb04\tfsbuild_TDC5DFC1BLD24_2020-04-10.11_47_23.coverage, but it was copied over to d:\_wf\06\13\TestResults\tests\tfsbuild_TDC5DFC1BLD24_2020-04-10_11_47_29\In\TDC5DFC1BLD24\tfsbuild_TDC5DFC1BLD24_2020-04-10.11_47_23.coverage

My mistake was to only delete the original location. When deleted in both places, the Code Coverage page became as expected - no more useless hyperlink, yes to the actual coverage.

The corrected powershell task in the YAML looks like this:

- powershell: |
    $cc = "$(Get-NuGetPackageBaseFolder Microsoft.CodeCoverage)\build\netstandard1.0\CodeCoverage\CodeCoverage.exe"
    $BinaryCoverageFile = Get-Item "$(Common.TestResultsDirectory)\tests\*\*.coverage"
    & $cc analyze /output:$(Common.TestResultsDirectory)\vstest-coverage.xml $BinaryCoverageFile.FullName
    Remove-Item (Get-ChildItem -Path "$(Common.TestResultsDirectory)\tests" -Recurse -Filter $BinaryCoverageFile.Name).FullName
  displayName: Convert Coverage Result To Xml

And of course, the Publish Test Results task must be moved after this task.

score 0 · Answer 2 · answered May 19 '22 at 03:17

I solved the problem by deleting the code coverage files after reportgenerator.

Here are the steps

steps:
  - task: DotNetCoreCLI@2
    displayName: dotnet test
    inputs:
      command: 'test'
      projects: '**/*.Tests.csproj'
      arguments: '--no-build --no-restore --configuration $(BuildConfiguration) --collect "XPlat Code Coverage" --settings CodeCoverage.runsettings --logger trx --results-directory $(Build.SourcesDirectory)/out/TestResults/' 
      publishTestResults: false

  - script: |
        dotnet tool install -g dotnet-reportgenerator-globaltool
        reportgenerator -reports:$(Build.SourcesDirectory)/out/TestResults/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/out/CoverageResults "-reporttypes:HtmlInline_AzurePipelines;Cobertura"
    displayName: Create code coverage report

  - task: DeleteFiles@1
    displayName: 'Delete coverage files'
    inputs:
      sourceFolder: $(Build.SourcesDirectory)/out/TestResults
      contents: | 
        \**\*.coverage
        
  - task: PublishTestResults@2
    displayName: 'Publish test results'
    inputs:
      testResultsFormat: VSTest
      testResultsFiles: '$(Build.SourcesDirectory)\out\TestResults\**\*.trx'
      failTaskOnFailedTests: true

  # Publish the combined code coverage to the pipeline
  - task: PublishCodeCoverageResults@1
    displayName: 'Publish code coverage report'
    inputs:
      codeCoverageTool: 'Cobertura'
      summaryFileLocation: '$(Build.SourcesDirectory)/out/CoverageResults/Cobertura.xml'
      reportDirectory: '$(Build.SourcesDirectory)/out/CoverageResults'

In my answer I run `Remove-Item (Get-ChildItem -Path "$(Common.TestResultsDirectory)\tests" -Recurse -Filter $BinaryCoverageFile.Name).FullName` at the end. It delets all the coverage files. — mark, May 19 '22 at 14:08
Hi @Xavier, would you please add some code for .net framework VS test — sebu, May 29 '23 at 08:16

How to publish Cobertura coverage report from the VS coverage produced by dotnet test --collect "Code coverage"?

2 Answers2