0

I have a flapping test in my test suite - it fails once out of every hundred runs or so.

However, this is kind of expected behavior as it's a test for a algorithm that runs a lot of things and returns a result - every once in while a result just can't be found. If I run the test suite again, everything is green.

Is there a way to re-run this one particular test if it fails and use the results of the rerun?

For example, 99% of the time this test passes. In the event that it fails, it should automatically be re-run without triggering a failure. If that re-run test fails, it should be reported as a normal failing test.

I'm thinking something like this, which is used to stop the test suite on failure: 

class << MiniTest::Unit.runner; self; end.class_eval do
  def puke(suite, test, e)
    super(suite, test, e)
    if ENV['FAIL_FAST'] && !e.kind_of?(MiniTest::Skip)
      turn_reporter.io.puts "****** Fail fast active for tests, exiting (environment variable FAIL_FAST exists)"
      exit(1)
    end
  end
end

but instead of 'failing fast' it 're-runs' the failing test. The difference is that the above code has an environment variable set - so if any test fails, it 'pukes' - I would only want to put this 're-run' flag on one specific test.

Has anyone found a way to do this?

dax
  • 10,779
  • 8
  • 51
  • 86
  • I understand your question and not sure I can help but why does the test fail? The purpose of tests is to have strict expectations that should always pass with a controlled set of data. – engineersmnky Sep 17 '14 at 13:25
  • @engineersmnky, you're right. I should have been more specific - I have unit tests that cover each individual component, the test in question is part of a group of acceptance/end-to-end tests. In this case, I feel confident that the component itself is working because of the unit tests and the surrounding acceptance tests - it's just this one test that occasionally fails (which can occur without issue in the running application) – dax Sep 17 '14 at 19:33

0 Answers0