How can I properly generate both TRX files and code coverage results with one call to "dotnet test"?

Question

First, some context: I have a Visual Studio solution containing several production class libraries and nine unit test projects. All the projects are targeting .NET 5. I am running .NET 5.0.401. All unit test projects have references to coverlet.collector. I do not use the older coverlet.msbuild package. From my reading, using XPlat Code Coverage is now idiomatic for .NET Core.

I have an Azure Pipelines pipeline that I use for CI builds. As a part of this pipeline, I want to run all the unit tests, generate unit test results for upload to Pipelines, and generate code coverage results for upload to Pipelines. I've already read several blog articles and documentation that would seem to make this simple; however, I've found it's anything but.

Let's start simple by running some commands on my local workstation. For the purposes of this exercise we will assume I have already successfully built the solution using this command:

dotnet build Solution.sln --configuration Debug

If I run tests like this:

dotnet test Solution.sln --configuration Debug --no-build --no-restore --collect:"XPlat Code Coverage" --results-directory artifacts/test-results

then I see code coverage results stored in subdirectories named after GUIDs. Why Microsoft decided to do this I have no idea, but that's out of my control.

(After running this command and taking the screenshot, I deleted the test-results directory.)

Remember that I also want to generate unit test results. To do that, I add the --logger trx parameter to the same command line. This time, lots of other folders are created that contain what appear to be duplicate code coverage results. Additionally, I get the .trx files I'm looking for.

You'll notice there are an additional nine code coverage reports in addition to the nine .trx files I want.

Similar duplicate files are generated on my self-hosted Pipelines build agent. In Pipelines, if I try to execute the following task:

- task: 'PublishCodeCoverageResults@1'
  displayName: 'Publish code coverage results'
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: 'artifacts/test-results/**/coverage.cobertura.xml'

I get an error:

##[warning]Multiple file or directory matches were found. Using the first match: C:\agent\_work\38\s\artifacts\test-results\$redacted\In\redacted\coverage.cobertura.xml
##[error]No code coverage results were found to publish.

The directory it's reporting here is not one of the GUID directories generated when I omit --logger trx but instead one of the duplicate directories.

This answer seems to indicate I'm doing everything properly. I've also posted a comment on that answer hoping for some assistance.

I have several questions:

1. Am I invoking dotnet test correctly in order to generate code coverage results and .trx files at the same time?
2. Does dotnet test support this scenario?
3. Is the function of the ReportGenerator extension in the Azure Pipelines marketplace actually built into dotnet now?
4. Which code coverage results are the "real" ones? Do I need to copy the GUID folders to another location before publishing code coverage results?
5. I want to specify a RunSettings file in order to do things like exclude members with certain attributes from being considered by the code coverage engine. Is RunSettings compatible with my workflow?

Answer 1

If you have multiple coverage files, you need merge them first before you publish. You may achieve this with following:

  - task: DotNetCoreCLI@2
    displayName: "dotnet test"
    inputs:
      command: test
      projects: "*.sln"
      publishTestResults: true
      arguments: -c Release --no-restore --no-build /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura

  - task: reportgenerator@4
    displayName: "Merge code coverage reports"
    inputs:
      reports: "**/coverage.cobertura.xml"
      targetdir: "$(Build.ArtifactStagingDirectory)/coverlet"
      reporttypes: "Cobertura"
      verbosity: "Verbose"

  - task: PublishCodeCoverageResults@1
    displayName: "Publish code coverage results"
    inputs:
      codeCoverageTool: Cobertura
      summaryFileLocation: "$(Build.ArtifactStagingDirectory)/coverlet/Cobertura.xml"

Note: some parameters probably need to be adjusted to your porject (like directories where you expect to have cobertura files).

I don't have in my code specifig logger for trx and I see in logs that they are produced.

If you want to pass runsettings please check this doc - Configure a test run.