4

We have around 100 test cases for our system. We are trying to build an automated test suite for it.

Say while running the tests the 25th test fails. Should our automated test system bail out here and stop execution, or should it just mark this as failed and continue trying to execute test cases 26th onwards (that is every test cycle will execute all 100 test cases irrespective of any failed test cases).

Ofcourse after a failed test case(for example no 25) if the system needs to be reset to execute test cases 26 onwards it will be taken care of.

Thanks

James

bezmax
  • 25,562
  • 10
  • 53
  • 84
James
  • 57
  • 2

2 Answers2

8

If your tests are independent - you should finish all of them. This way you can monitor the system stability and see all of the problems at once without re-running tests countless times.

bezmax
  • 25,562
  • 10
  • 53
  • 84
0

If this is running without human intervention, say as part of some automated build, I would want to attempt all tests.

However, there are scenarios where you're in the mode of fixing problems where it might save a human's time to just stop. If it's easy I'd like to offer a "Stop on first failure" option.

djna
  • 54,992
  • 14
  • 74
  • 117
  • Could you give any example of such scenario? I can't think of any. – bezmax Jul 15 '11 at 07:30
  • @Max: I can think of one for the case where you can't start a new test until the old one finishes. If your test suite normally takes an hour to run and you're making a quick 5 minute fix, you don't want to have to wait an hour to re-run your test when you're trying to fix the bug with test #2. – Bryan Oakley Jul 15 '11 at 11:19
  • @Bryan Oakley: When the tests are dependent on eachother - it is common practice to not separate them. They should be contained in 1 test block/method/unit/etc. Therefore, If you have `test1()`, `test2and3and4()` and `test5()`, on first execution `test1()` and `test5()` will pass and `test2and3and4()` will fail. Then you fix and rerun everything - and `test2and3and4()` also pass. – bezmax Jul 15 '11 at 11:25