2

I use Azure DevOps Server 2020 with self hosted agents and created a CI pipeline which executes all tests in parallel on one agent. The ~5000 tests (no UI tests) need around 7min to complete. That's way to slow for our needs, so to speed things up I added 3 agents, put the VsTest task into another job in the same pipeline with parallel: 4. All 4 agents first download the build artifacts and run a slice of the tests. Unfortunately this actually made it worse. Now the test run needs around 8 min on each agent.

My vstest yaml for 1 agent

- task: VSTest@2
  displayName: 'Run tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2:
            **\*test*.dll
            !**\*TestAdapter.dll
            !**\*TestFramework.dll
            !**\obj\**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: false
      rerunFailedTests: false

My vstest yaml for 4 agents

- task: VSTest@2
  displayName: 'Run tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2:
            **\*test*.dll
            !**\*TestAdapter.dll
            !**\*TestFramework.dll
            !**\obj\**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: false
      distributionBatchType: 'basedOnExecutionTime'
      rerunFailedTests: false

I even tried batching by assembly and based on number of tests + number of agents, test run time still sits at ~8min.

Comparing this to our old UI based CI pipeline, with multi-config and multiplier on a variable with 4 TestCategories, which runs even more tests ~10000 (including the 5000 of the new pipeline) but these are distributed by TestCategory on the same 4 agents (Cat1 on agent1, Cat2 on agent2 and so on), the agents average on ~5min each.

The yaml of the UI based one looks like this:

steps:
- task: VSTest@2
  displayName: 'Run tests'
  inputs:
    searchFolder: '$(Build.BinariesDirectory)'
    testFiltercriteria: 'TestCategory=$(Tests)'
    runInParallel: true
    codeCoverageEnabled: false

I think I have to be missing something obvious.

Thanks in advance!

Edit 1:

I connected to my agents with RDP and inside the task manager there are multiple instances of testhost.x86 running, up to 8 simultaneously but not constantly. If I start my tests locally the 8+ instances of testhost.x86 are up almost all the time and rarely vanish at all. If that's any help.

bego
  • 99
  • 3
  • 9
  • I don't have a solution for you, but just letting you know I had the same issue on previous version. I tried to find developercommunity I logged about it, but may have been deleted. My guess at the time was that when you run single it uses all the cores; when you run in parallel machines it only uses a single core. So you don't really gain much and lose because of the test discovery and splitting. I think we ended up just increasing the core count on those few machines :) – Matt Oct 15 '20 at 14:00
  • Hi, "good" to hear, I'm not alone with this. These 4 agents are i7 with 16GB RAM and 250GB SSDs. Would be bothersome to increase the cpu count on these haha – bego Oct 15 '20 at 14:13

0 Answers0