2

My issue seems to be very similar to: rspec test passes in isolation, but fails when run with other tests - but the accepted answer is something I'm already doing: I'm using avdi's database cleaner setup (and I've tried deletion instead of truncation like thoughtbot uses in their suspenders example app): http://devblog.avdi.org/2012/08/31/configuring-database_cleaner-with-rails-rspec-capybara-and-selenium/

So the account record from this code:

let(:account) { create(:account) }

seems to be being deleted. I'm hitting a page not found because the account doesn't exist. So switching back and forth between the different database cleaner strategies seems to be causing the issue?

I can literally run rspec --only-failures and all my tests that just failed will pass. Its only happening with feature/capybara specs.

EDIT: Here's my config: https://gist.github.com/gregblass/b886f79b0d8e2e2015af

UPDATE: Here are some examples of failing tests: https://gist.github.com/gregblass/1b878d92a2b9dad355e0

UPDATE 2: It happening after I do a js: true test. Something with those are screwing up my other non-js capybara tests that follow it.

Community
  • 1
  • 1
Greg Blass
  • 3,523
  • 29
  • 42
  • Use the "RSpec with Capybara Example"' configuration from the database_cleaner README - https://github.com/DatabaseCleaner/database_cleaner - the append_after helps fix some flaky errors. If that doesn't fix it please post any othe RSpec config you are using, and example of the exact error message you're getting – Thomas Walpole Mar 18 '16 at 02:47
  • That doesn't seem to make any difference. I'll throw my RSpec config up in a gist or something tomorrow...can't stay awake any longer. – Greg Blass Mar 18 '16 at 02:58
  • I added a gist of my config. – Greg Blass Mar 18 '16 at 10:46
  • We still need the exact error. The next thing I'd do is to make a minimal spec (two or even one RSpec example(s)) that reproduces the problem and look at the SQL in test.log to see whether anything is deleting accounts and if so whether it's your code or database_cleaner. – Dave Schweisguth Mar 18 '16 at 14:16
  • Can you provide an example of a test that fails? – Thomas Walpole Mar 18 '16 at 18:18
  • Added two examples of some failing tests above. – Greg Blass Mar 18 '16 at 19:07
  • None of the tests you added are js: true, so is it only non-JS tests that fail, and only if JS tests have run before them? and do all the tests you posted fail? or only some of them? - Also as requested previously - please post an exact error message you get. – Thomas Walpole Mar 18 '16 at 19:31
  • I thought that originally but I don't think that is actually the case. I can't pin it down that specifically. – Greg Blass Mar 18 '16 at 19:32
  • So you do have failing JS tests? – Thomas Walpole Mar 18 '16 at 19:33
  • Yes, I do. Both JS and non-js will fail because the account can't be found. – Greg Blass Mar 18 '16 at 19:34
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/106759/discussion-between-gregblass-and-tom-walpole). – Greg Blass Mar 18 '16 at 19:39

1 Answers1

4

I was able to use rspec --bisect and pin down the culprits - they were JS capybara tests. After I ran JS tests, the database cleaner switching strategy that everyone seems to recommend is deleting my accounts that I'm declaring in my non-js capybara tests that come after.

This is one workaround:

http://stefan.magnuson.co/articles/rails/robust-integration-testing-in-rails-4-with-rspec-capybara-and-selenium/

Many tutorials and StackOverflow answers out there have JS and non-JS tests mixed together, with the strategy being switched between transaction and truncation on each test.

In my experience this has resulted in database access race conditions that have caused otherwise well-written, independent tests to fail intermittently.

The solution I have settled on is to run all non-JS tests first (shuffled), and then all JS tests (also shuffled). This allows for discovery of tests that incorrectly expect certain state, or fail to clean up after themselves (by virtue of the random execution order), while not attempting to freely mix JS and non-JS tests. As these different classes of test have different purposes, I see no disadvantage in this approach, with the exception of it being somewhat non-standard.

Another is to change your database cleaner configuration to the following (taken from @ryanbigg 's multitenancy with rails book):

config.before(:suite) do
  DatabaseCleaner.strategy = :truncation
  DatabaseCleaner.clean_with(:truncation)
end

config.around(:each) do |example|
  DatabaseCleaner.cleaning do
    example.run
  end
end

This is slower, but I can't even really tell much of a difference on my Macbook Pro 16GB/SSD.

Thank you @tomwalpole for all your help, @barelyknown for suggesting rspec --bisect, and @ryanbigg for your great book on testing multitenancy with capybara with good working code examples, and Stefan Magnuson for that article (who I can't find a SO handle for).

Greg Blass
  • 3,523
  • 29
  • 42