4

I am working in a small team who have just moved to Azure Dev-Ops. When a new build is pushed out we can't seem to figure out a seamless and convenient way to both run regression tests and have full historical data of the test plans which have been run. I am worried that once Azure DevOps has been in use for a while, it may become difficult to locate older test-runs.

So far we have tried creating a large test-suite, with sub-test suite folders for different areas of functionality. Each sub-test suite is then individually run in Test-Runner. The problem with this is when we reset the tests and run them again, we lose historical test data, and any tests that aren't completed on the test run go missing, making it look like a higher percentage of test cases have passed/failed. Another problem is that test-plans are fragmented.

Another option we have explored is adding a label (instead of sub-test suite) to each test case. This causes a problem when trying to run all the tests as Test-Runner appears to have a limit of no more then 100 test cases being run at once. On top of this the test-runs name is indistinguishable from each other when ran in separation, as they all have the same name (from the test-suite).

An ideal solution would be something like TestRail (which we have just moved from) where cases from a test-suite can be selected for a run, and then the test-plan is stored indefinitely. Unfortunately we are unable to move back to TestRail.

SteveMcKenna
  • 41
  • 1
  • 3

1 Answers1

1

In the Azure DevOps Documentation under creating a Test Plan they put the test plan in for a specific sprint.

What you can do is create a Regression Test Area Path containing tests you want and then every time you want to run a regression test you can add this to the test Plan created in the Sprint.

It does seem a bit confusing but in essence is similar to how it's done in TestRail when you create a Test Run and select what tests you want in the run. It can be a bit of a terminology switch to create a Test Plan for every sprint, but you can do that and then add what regression tests you want to run during the sprint and that way you don't lose any historic test data.

Aad
  • 144
  • 1
  • 5
  • 1
    This CAN be done, but is it a good practice? – GaTechThomas Mar 25 '21 at 18:12
  • Using area path to mark test cases for regression is a bad idea. Much better is to use test case priorty (e.g. 1 or 1+2) and create a test plan with query-based test suites to pull those test cases for execution. – YMM Jan 25 '23 at 09:53