1

I'm setting up a TFS testing build and want to support full parallel testing. Only some of our tests run in parallel and therefore after they finished running on the build agent - the build agent do not getting new tests to test. Therefore, a build that took us 30 minutes start to take an hour and more because all the renaming tests running on only one build agent. All the other 20 build agents stop to run after several minutes because they finished running their bulk of tests. We run our parallel tests with MSTest2. Our RunSettings.runSettings declare to run as parallel and are correct.

We tried different Advanced Execution Options on the build. We tried based on past running time of tests. based on number of tests and agents and based on test assemblies. None of them work - after the agent finished his bulk of tests he just go back to idle. We tried giving the machine in the build to get only one test each bulk and it was indeed faster but we lost the privilege of parallel testing on each machine.

We want our build to run on this way: We start it, and the tests start to run. All the machines will get count of tests to run. When the build agent finish to run all the tests he was given, it will get tests from a machine that didn't finish to run his tests. This way we'll enjoy parallel testing build and parallel tests on a machine. We don't want to get to the point where our build has 2 steps - 1 step that run the tests parallel and another one that run our tests which are not parallel.

Does it possible to create a build that answer all of the above? We started to lose our heads because of this and we didn't find examples to builds that will do as we want. Or how at least change the settings to prevent from build agent to stop getting new tests after he finished his bulk.

Best regards and I hope that I made myself clear enough.

jessehouwing
  • 106,458
  • 22
  • 256
  • 341
Noa Sasson
  • 23
  • 4

1 Answers1

0

No this is not possible. The tests are distributed at the start of the test run. What you can do is the following:

  • Make sure all your slow tests are marked using a [TestCategory("slow")] attribute.
  • Create one Run Tests task in Azure Pipelines and use the filter to run all tests except slow tests. This will distribute the remaining tests across all agents.
  • Create one Run Tests task in Azure Pipelines and use the filter to only run the slow tests. This will distribute all slow tests to all agents

You end up with two test runs, but your tests will be more equally distributed.

As an added advantage your slow tests can optionally be skipped if one of your fast tests already fail.

Basically what I blogged here:

jessehouwing
  • 106,458
  • 22
  • 256
  • 341
  • A little bit sad to hear it but this is what I was afraid off. What I still do not understand is why build agents stop running more tests from the list and go to Idle. This is another thing I need to prevent. – Noa Sasson Dec 28 '18 at 13:57
  • The tests are simply distributed at the start of the job and there is no communication between the agents or the agent and the server during execution. It simplifies things considerably and works with every technology out there. – jessehouwing Dec 28 '18 at 13:59
  • Yeah but this is a little bit weird because that way you actually lose time because agent will stop working before all the others (this way the time that take for the tests to run will increase). No? – Noa Sasson Dec 28 '18 at 14:45
  • If you categorize your tests correctly you will get 2 test phases which will be distributed as best as possible and will keep all agents as busy as possible. With a little overhead for setting up the second phase. The best solution is to refactor those slow tests to be fast ;0). – jessehouwing Dec 28 '18 at 14:53
  • If that would be easy I would have done it before ;) By the way do you know if the builds leave any temp files in the machine? Not all of our machines you a lot of place and I am worried that they'll crash because of lost of place – Noa Sasson Dec 28 '18 at 15:12
  • There is no framework that support sync between the agents? – Noa Sasson Dec 28 '18 at 15:14
  • No. There is no framework that supports syncing agents. – jessehouwing Dec 28 '18 at 16:25
  • Does it possible to run diffrent phases in parallel? – Noa Sasson Jan 01 '19 at 08:46
  • Yes. You can configure the dependencies between phases in such away that they both start at the same time. But if they're targeting the same test agents, they may simply be waiting still... – jessehouwing Jan 01 '19 at 10:17