2

Even though I am aware of the fact that tests should run reliably, my experience tells me: that cannot always be accomplished with reasonable effort (and need not be accomplished; see my calculation below).

Particularly, if tests are introduced for a pre-existing Web application that needs to be persistently improved, it can be difficult to build reliable E2E tests. But, in contrast, it is pretty easy and besides sufficient to build tests which crash occasionally (as long as they fail reliably when it comes to expects/asserts).

If you are using Protractor for E2E testing, you may have experienced that as well.

Statistics tell me that a test that is known to have a 25% chance to crash, will have a 6.25% chance to crash twice when run twice, a 1.56% chance to crash three times when run three times, a 0.39% chance to crash four times when run four times and a 0.10% chance to crash five times when run five times (and so on).

Hence, running a set of such tests until each of them manages to terminate without error is no big deal.

My demand is to run a Cucumber.js scenario again and again until it succeeds for the first time during a single feature run, and then take the test pass/fail result of the succeeding run.

I tried to build an After hook that re-runs the scenario but I did not find a way of invoking a scenario from within the hook. Apart from that, I did not find a way of discarding a crashing scenario run.

Please tell me which options I have. I am using Grunt, Protractor and Cucumber.

Since our test suite is huge, still growing rapidly and run by an automated build process, I need a way of running unreliable tests as reliably as possibly.

ideaboxer
  • 3,863
  • 8
  • 43
  • 62
  • I think this is going to lead to an unmanageable test suite. How will you distinguish real failures from failures due to flakiness? You'll only be able to run each test a fixed number of times and then give up, or your suite will run forever. My experience is that you should retry tests as little as possible, because doing so makes it easier for developers to write flaky tests. – Dave Schweisguth Apr 29 '16 at 19:07
  • "How will you distinguish real failures from failures due to flakiness?" If I make sure the test never passes if the tested conditions are not true (by taking care when it comes to the "Then" step of a scenario and making the "Then" step **reliably not pass if the app does something tested wrong**), it is easy to distinguish: As soon as a test _never_ manages to pass, I need to inspect the tested scenario manually. **If, on the other hand, this is not the case, I can be (as) sure (as if I ran a never-crashing test) that everything is alright with the scenario.** – ideaboxer May 03 '16 at 20:52
  • "You'll only be able to run each test a fixed number of times and then give up, or your suite will run forever." Giving up after automatically trying it 10 times (max) is not a big deal: You can always run a scenario manually and see if you can reproduce the unexpected behavior. Since I know that usually each test does not crash in _each_ run when being run twice or three times, that will be absolutely fine for me. Having an occasionally crashing but still reliably not-passing-on-app's-misbehavior test is always better than having no test at all. – ideaboxer May 03 '16 at 20:56
  • "My experience is that you should retry tests as little as possible, because doing so makes it easier for developers to write flaky tests." I agree with you 100%. But still: having one or more somewhat flaky tests is much better than having no tests at all. – ideaboxer May 03 '16 at 21:04

1 Answers1

0

A common reason for angular failures is not waiting for all the pending requests to complete. In my page constructors I insert a wait for this Javascript to return true:

"return angular.element(document.body).injector().get(\'$http\').pendingRequests.length == 0;"

before proceeding. By being careful to explicitly wait for JavaScript to complete before continuing with my tests their reliability is quite high.

That being said, there are rerun frameworks out there. Here is one by Nickolay Kolesnik .

MikeJRamsey56
  • 2,779
  • 1
  • 20
  • 34
  • Unfortunately, there aren't any rerun frameworks out there for Cucumber+Protractor (which is JavaScript based). – ideaboxer May 03 '16 at 20:44