1

I am running a few ordered tests using Azure DevOps pipelines. The result for the tests is grouped into a single ordered test. It is becoming very difficult to interpret the result. Is it possible to obtain a separate pass/fail result?

In the screenshot below, the ordered test contains about 75 tests and only one of them fails. But that results in an overall 1 Test failed result. Is it possible to show 74 passed, 1 failed result in the Tests Results?

Test result

Zankhana Rana
  • 169
  • 2
  • 16

2 Answers2

2

According to your description, since you are using ordered test. It's an expect behavior. The tests will be grouped and be reported in a summarized view.

You could take a look at the statement in our official doc here-- View summarized test results

During test execution, a test might spawn multiple instances or tests that contribute to the overall outcome. Some examples are, tests that are rerun, tests composed of an ordered combination of other tests (ordered tests) or tests having different instances based on an input parameter (data driven tests).

As these tests are related, they must be reported together with the overall outcome derived from the individual instances or tests. These test results are reported as a summarized test result in the Tests tab:

  • Rerun failed tests: .......
  • Data driven tests: ...... The summarized view is also available for ordered tests (.orderedtest in Visual Studio).

It's not possible to show 74 passed, 1 failed result in the Tests Results.

To see test results of individual tests in the ordered test suite you may need to download the *.trx file and open it in Visual Studio.

PatrickLu-MSFT
  • 49,478
  • 5
  • 35
  • 62
  • I am using the *.trx file to view the results on visual studio. Isn't it even possible to publish separated result using the .trx file? – Zankhana Rana Feb 26 '20 at 03:42
  • @ZankhanaRana Sorry, not sure about this. I haven't run into this scenario, you could give a try with using a sperate [Publish Test Results task](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml) to check if this do the trick. If this also not work, afraid no way to handle it at the moment. – PatrickLu-MSFT Feb 26 '20 at 03:48
  • 1
    I followed the steps in the link you mentioned but no luck. Thank you for the response. – Zankhana Rana Feb 26 '20 at 03:55
  • 1
    @ZankhanaRana Thanks for your quick update. Afraid this is a design limitation. You may have to keep using the *.trx file to view the results on visual studio to see pass test results. Apologize for any inconvenience. Besides, if my reply helped or gave a right direction. Appreciate for marking it as an answer, which will also help others in the community. – PatrickLu-MSFT Feb 26 '20 at 04:00
  • Let me wait for some more time if there is any chance someone has come across the problem and has some solution to it. Otherwise I will mark this as the answer – Zankhana Rana Feb 26 '20 at 04:02
1

This is what I did to get the individual results-

All the tests I wanted to run were in a single ordered test. I changed the build definition to run them individually and not as a part of a single ordered test. The only drawback of following this approach is we cannot rerun the failed tests.

Test files

Result:

Results

Zankhana Rana
  • 169
  • 2
  • 16