1

Our codebase has more than 100 projects with tests in each. Some test assemblies take more time while some other assemblies are taking less time for the execution of the tests.

The Azure DevOps Server is running our whole test suit in parallel, which makes it really fast. But the problem is, that the long running tests are started in the middle of the testrun, which has the effect, that the whole testrun will be longer.

Is there a way, to influence the order of how and when the test assemblies are started? I want to start the long running test assemblies first and after that the fast test assemblies.

F. Baum
  • 301
  • 1
  • 5
  • 17

2 Answers2

0

Since you are running the Test in parallel, you could try to use the Based on past running time of tests option in Visual Studio Test task.

test task settings

According to this doc about Parallel test:

This setting considers past running times to create slices of tests so that each slice has approximately the same running time. Short-running tests will be batched together, while long-running tests will be allocated to separate slices.

This option allows tests to be run in groups based on running time. Finally , each group will be completed in a similar time.

Hope this helps.

Kevin Lu-MSFT
  • 20,786
  • 3
  • 19
  • 28
  • I thought about it too, but don't I need multiple agent for this? – F. Baum Jun 12 '20 at 08:34
  • @F.Baum Yes. You need to configure multiple agents. Based on my test, the working hours of each job are similar. – Kevin Lu-MSFT Jun 12 '20 at 08:39
  • As of several reasons, I'm not able to use multiple agents. Is there another way to order the execution? – F. Baum Jun 14 '20 at 08:23
  • Could you please share the way the test runs in parallel? Do you use this option: `Run tests in parallel on multi-core machines`? – Kevin Lu-MSFT Jun 15 '20 at 08:33
  • Yes. I'm using the flag `Run tests in parallel on multi-core machines` – F. Baum Jun 16 '20 at 08:53
  • Check this [doc](https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops#massively-parallel-testing-by-combining-parallel-pipeline-jobs-with-parallel-test-execution). If the parallel tests run on a single machine , it could also use the parallelism features. You could try it and check if it could work. – Kevin Lu-MSFT Jun 18 '20 at 08:46
0

We have achieved this by arranging the project-folders so they sort to give the longest running test assemblies first. You can see the order that VSTest finds the assemblies in the Azure DevOps output. From there you can rename folder to affect the order.

It would be nice if there was another way to effect this.

Rob Dal Santo
  • 31
  • 1
  • 5