0

I have a pipeline with a "Visual Studio Test" task where "Minimum # of tests" is set to 10:

test task minimum number of tests configuration

However, when a release does not meet the 10 minimum tests requirements, the right error is included in the logs:

2021-02-18T04:47:15.4609022Z ##[error]The specified minimum number of tests 10 were not executed in the test run.

azure devops test results

However, there two lines below, there us the following log:

2021-02-18T04:47:16.2005970Z Execution Result Code 1 is non zero, checking for failed results
2021-02-18T04:47:17.8073253Z Failed tests if any were marked as flaky hence changing the exit code to 0

And the task is marked as successful. None of the tests in the entire test suite is marked as flaky? Where can the Failed tests if any were marked as flaky hence changing the exit code to 0 log coming from?

agent job logs

kiewic
  • 15,852
  • 13
  • 78
  • 101
  • Not get your response for several days, would you please share your latest information about this issue? If you have any concern, feel free to share it here. – Hugh Lin Feb 24 '21 at 09:32
  • Hi Hugh, we are going to try disabling "Flaky test detection" but we need to confirm with other teams that nobody is relying on it first. So, this may take some time before we can disable. – kiewic Feb 25 '21 at 22:08
  • @HughLin, I finally confirmed that disabling **Flaky test detection** gets rid of the **Failed tests if any were marked as flaky hence changing the exit code to 0** issue and correctly marks the release as failed – kiewic Jun 16 '21 at 22:09

1 Answers1

1
Failed tests if any were marked as flaky hence changing the exit code to 0

You can try to disable Flaky test detection in the Test management of project settings.

enter image description here

Flaky tests present a barrier to finding real problems, since the failures often don't relate to the changes being tested. A flaky test is a test that provides different outcomes, such as pass or fail, even when there are no changes in the source code or execution environment. Flaky tests also impact the quality of shipped code.

For details , please refer to this document.

Hugh Lin
  • 17,829
  • 2
  • 21
  • 25