7

We have intermittently failing tests due to Net::ReadTimeout errors.

We have yet to figure out a permanent fix. For right now we want to try rescuing that specific error and re-running the test. Not a ideal solution or true fix but we need something in the short term.

How can we re-run the rspec test that failed? I mean how can the test suite do this automatically for that test?

We can catch the error like this:

# spec/support/capybara.rb
def rescue_net_read_timeout(max_tries = 1, tries = 0, &block)
  yield
rescue Net::ReadTimeout => e
  Rails.logger.error e.message
end

but how do we make it try re-running that test? We want to try re-running the test and then if the re-run passes move on with no error (ideally log it though), else fail for real and consider the test and hence the suite to have failed.

Stefan
  • 109,145
  • 14
  • 143
  • 218
Michael Durrant
  • 93,410
  • 97
  • 333
  • 497
  • Do you get timeouts accessing your own application or an external service? – ndnenkov Dec 13 '16 at 16:09
  • If you are getting `Net::ReadTimeout` from external services/API's I would recommend stubbing them. Webmock https://github.com/bblimke/webmock is brilliant for this. – DMH Dec 13 '16 at 16:10

2 Answers2

3

The following method uses the wait gem to retry a test - or a portion of a test, possibly just an assertion. It 

def retry_test(&block)
  Wait.new(delay: 0.5, attempts: 10).until do
    begin
      yield    
      true # test passed
    rescue RSpec::Expectations::ExpectationNotMetError => x
      # make sure it failed for the expected reason
      expect(x.to_s).to match /Net::ReadTimeout/
      false # test failed, will retry
    end
  end
end

One can verify why it failed, and report a failure immediately if ever the test failed for another reason.

it 'should get a response' do
  retry_test do
    # The entire test, or just the portion to retry
  end
end
philant
  • 34,748
  • 11
  • 69
  • 112
2

I use a ruby gem called rspec-repeat. Out of a few hundred automated tests, we might run into a flaky one here or there and this helps us get past false negatives.

Ideally it's best to continue to diagnose these flaky tests, but this helps alleviate issues in the interim.

Side note, rspec-repeat based off of a another library named rspec-retry, but I find the code base for this one to be tidier and the configuration easier to use.

Phil
  • 886
  • 7
  • 20