Generating Runsettings configuration file during VSTS Build Task "Visual Studio Test"

Question

To start my main issue is seeing differing code coverage outputs between visual studios 2017, and VSTS (Azure Dev Ops) build pipeline task "Visual Studios Test".

When analyzing the code coverage in visual studios, I am using 2017, for my solution's tests it seems during the process it is creating a temporary runsettings configuration file on the fly and using it when calling the vstest.console.exe process.

What visual studios is doing is analyzing our solution and creating specific configuration that filters out dependency dlls from being included in our code coverage report that gets generated. So in visual studios if I look in the output window for tests I will see the temp runsettings file that gets generated.

<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector uri="datacollector://microsoft/unittestisolation/1.0" assemblyQualifiedName="Microsoft.VisualStudio.TraceCollector.UnitTestIsolationDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" friendlyName="UnitTestIsolation">
    <Configuration>
    </Configuration>
  </DataCollector>
  <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector">
    <Configuration>
      <CoverageFileName>Coverage 2019-06-11 14_30_31.coverage</CoverageFileName>
      <CodeCoverage>
        <ModulePaths>
          <Exclude>
            <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
          </Exclude>
          <Include>
            <ModulePath>.*\\<MyNamespace>\.API\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.ACL\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Services\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.API\.Contracts\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Services\.Contracts\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.ACL\.Contracts\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Services\.UnitTests\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.ACL\.UnitTests\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.PaymentSystem\.Util\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.MerchantContracts\.Clover\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.UnitTests\.Data\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Data\.EF\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Domain\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Domain\.UnitTests\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Core\.NuGetResources\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.MerchantContracts\.Mercury\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Processor\.exe</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Processor\.Testing\.exe</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.Processor\.UnitTests\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.MerchantContracts\.Mindbody\.dll</ModulePath>
            <ModulePath>.*\\<MyNamespace>\.API\.UnitTests\.dll</ModulePath>
          </Include>
        </ModulePaths>
    ... etc
</RunSettings>

This results in the following being pull, note don't try matching these up with the above, this is only meant to give you a idea on whats happening. These are all correct.

<MyNamespace>.infrastructure.libraries.dll
<MyNamespace>.infrastructure.libraries.domain.dll
<MyNamespace>.infrastructure.libraries.logging.dll
<MyNamespace>.infrastructure.libraries.webapi.dll
<MyNamespace>.acl.dll
<MyNamespace>.acl.unittests.dll
<MyNamespace>.api.contracts.dll
<MyNamespace>.domain.dll
<MyNamespace>.domain.unittests.dll
<MyNamespace>.merchantcontracts.clover.dll
<MyNamespace>.merchantcontracts.mercury.dll
<MyNamespace>.merchantcontracts.mindbody.dll
<MyNamespace>.processor.unittests.dll
<MyNamespace>.services.contracts.dll
<MyNamespace>.services.dll
<MyNamespace>.services.unittests.dll
<MyNamespace>.unittests.data.dll

Now say I create a blank runsettings and test the code coverage again in visual studios.

My runsettings.

<RunSettings>
<DataCollectionRunSettings>
    <DataCollectors>
        <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
            <Configuration>
                <CodeCoverage>
                </CodeCoverage>
            </Configuration>
        </DataCollector>
    </DataCollectors>
</DataCollectionRunSettings>
</RunSettings>

The code coverage that is generated matches the coverage from before but then we also get additional third party/unwanted dlls included.

fluentassertions.core.dll
fluentassertions.dll
moq.dll
nunit3.testadapter.dll

Now in VSTS, the "Visual Studio Test" task without any runsettings configuration is including the additional third party dlls listed above. Unlike Visual studios 2017, it isn't creating a runsettings configuration on the fly to filter them out. Instead I am having to create a runsettings configuration manually and include it in my repo just to use in the build pipeline task.

The main issue here is maintainability. In the long run I don't want to personally manage this file and have to remember to update it every time I add another project that needs to get pulled. I could make the module path more generic so it is pulling dlls base on a portion of the dll name.

<ModulePath>.*\\<MyNamespace>.*\.dll</ModulePath>

This still leaves room for error. It just seems silly to me that I need to manually configure this when somehow visual studios is doing this in the background automatically.

I was hoping that someone would know how to generate the code coverage in VSTS using the same process that visual studios does so that it is creating the runsettings configuration using the projects in the solution to filter out unwanted dll tests.

Generating Runsettings configuration file during VSTS Build Task "Visual Studio Test"

0 Answers0