2

When reading a question about testing a Java program several times with random seeds the word testing sparked of the association with unit testing in my mind, but that might not have been the kind of testing going on there.

In my opinion introducing randomness in a unit test would be considered bad practice, but then I started considering the case where a (small) percentage of failure might be acceptable for the moment.

For example, the code fails to pass the unit test once every 10^n for n > 3 and gradually you want n to go to infinity without the test going red, maybe yellowish.

Another example might be a system-wide test where most of the time things go right, but you still want to limit/know how often they might go wrong.

So my question is, are there any frameworks (full spectrum of testing) out there that can be persuaded to allow a percentage of failure in an huge/excessive amount of repeated tests?

Community
  • 1
  • 1
Gressie
  • 507
  • 2
  • 7
  • 1
    This might not be the answer to your question, but I guess what you might want to look at is [_Property Based Testing_](http://blog.jessitron.com/2013/04/property-based-testing-what-is-it.html), that can generate large amount of tests, and then return you the failing test case. – GKalnytskyi Feb 03 '16 at 03:58
  • @GKalnytskyi thanks for your input! That link made me remember a friday afternoon topic from some professor at university and it mentions the issues with Property Based Testing, getting the input right and not blowing up the JVM. A great solution for input to the accepted answer. – Gressie Feb 03 '16 at 17:49

1 Answers1

2

You can "persuade" most testing frameworks to allow partial failures by performing your "tests" not using the framework directly but just by doing plain old conditional (eg: if-statements) and recording the percentage of failures. Then use the framework to assert that this percentage is below your threshold.

If your tests are deterministic (which is generally considered to be a good thing) then another approach is to test for the current behavior even when it's wrong, but to comment the incorrect assertions (typically with what the "right" answer should be). If these tests ever fail you can check the comment. If the code has become "correct" then great, update that assertion. If not, then decide whether the change in behavior is better or worse than the old behavior, and act accordingly. This approach lets you sort of "tighten the screws" over time.

Laurence Gonsalves
  • 137,896
  • 35
  • 246
  • 299
  • Thanks, letting the framework test for percentages is a great idea! This should at least put people in the right mindset for solving similar problems, should it not be sufficient enough to completely solve it. – Gressie Apr 27 '11 at 06:35