Parallel execution of tests on TFS

Question

We use TFS on our project. I have set Parallelism -> Multi Agent in the phase settings. The command itself to run (.NET Core) is:

dotnet test --filter TestCategory="Mobile" --logger trx -m:1.

Do I understand correctly that these settings will not split the tests between the two agents, but run the command above on the two agents?

Your understanding is correct, this will run the same job on multiple agents. Useful for load testing and a few other scenarios. — jessehouwing, Jan 31 '22 at 12:21
And there is no way to use some similar option to distribute a single task among multiple agents? For example, running multiple tests in a single task which will distribute the tests across multiple agents rather than running multiple independent test runs. — Shep, Jan 31 '22 at 12:29
The Visual Studio Test task will detect and slice the workload when multiple agents are configured. Calling dotnet test from the commandline isn't clever enough to do that. Try using the vstest task instead. — jessehouwing, Jan 31 '22 at 12:34
Or create a custom slicer function, as can be seen in this sample repository: https://github.com/idubnori/ParallelTestingSample-dotnet-core — jessehouwing, Jan 31 '22 at 12:36

jessehouwing · Accepted Answer · 2022-01-31T14:55:49.607

The Visual Studio Test (- task: VSTest@2) has built-in magic to distribute the test based on configurable criteria:

You could switch to using the vstest task instead; to run your tests to get this "magic".

The dotnet core task or invoking dotnet straight from the command line doesn't have this magic.

There is a github repo that shows how to take advantage of the default of the hidden variables that are set by the agent when running in parallel:

#!/bin/bash

filterProperty="Name"

tests=$1
testCount=${#tests[@]}
totalAgents=$SYSTEM_TOTALJOBSINPHASE
agentNumber=$SYSTEM_JOBPOSITIONINPHASE

if [ $totalAgents -eq 0 ]; then totalAgents=1; fi
if [ -z "$agentNumber" ]; then agentNumber=1; fi

echo "Total agents: $totalAgents"
echo "Agent number: $agentNumber"
echo "Total tests: $testCount"

echo "Target tests:"
for ((i=$agentNumber; i <= $testCount;i=$((i+$totalAgents)))); do
targetTestName=${tests[$i -1]}
echo "$targetTestName"
filter+="|${filterProperty}=${targetTestName}"
done
filter=${filter#"|"}

echo "##vso[task.setvariable variable=targetTestsFilter]$filter"

This way you can slice the tasks in your pipeline:

  - bash: |
      tests=($(dotnet test . --no-build --list-tests | grep Test_))
      . 'create_slicing_filter_condition.sh' $tests
    displayName: 'Create slicing filter condition'
  
  - bash: |
      echo "Slicing filter condition: $(targetTestsFilter)"
    displayName: 'Echo slicing filter condition'
  - task: DotNetCoreCLI@2
    displayName: Test
    inputs:
      command: test
      projects: '**/*Tests/*Tests.csproj'
      arguments: '--no-build --filter "$(targetTestsFilter)"'

I'm not sure whether this will support 100.000's of tests. In that case you may have to break the list into batches and call dotnet test multiple times in a row. I couldn't find support for vstest playlists.

Parallel execution of tests on TFS

1 Answers1