Rspec tests pass when running individually but fails when running all together

Question

The project uses MongoDB, so this answers weren't helpfull: 1, 2, 3

I can successfully execute every test file, or even small groups of test files, but they fail when running all together. rspec spec/*

Finished in 23 minutes 45 seconds (files took 23.02 seconds to load)
2071 examples, 1357 failures, 28 pending

The project is a little bit old. MongoDB 3.6, Ruby 2.5.1 with Padrino as framework. Also, we use DatabaseCleaner and Fabrication. A lot of errors show the classic message of Sinatra doesn’t know this ditty.

61) BlogController POST :create Microsite request should return code 12 for invalid token
      Failure/Error: @response_body = JSON.parse(last_response.body)

      JSON::ParserError:
        784: unexpected token at '<!DOCTYPE html>
        <html>
        <head>
          <style type="text/css">
          body { text-align:center;font-family:helvetica,arial;font-size:22px;
            color:#888;margin:20px}
          #c {margin:0 auto;width:500px;text-align:left}
          </style>
        </head>
        <body>
          <h2>Sinatra doesn’t know this ditty.</h2>
          <img src='http://localhost.la/__sinatra__/404.png'>
          <div id="c">
            Try this:
            <pre># in app.rb
        class Microsite
          post &#x27;&#x2F;v1&#x2F;microsite&#x2F;blog&#x27; do
            &quot;Hello World&quot;
          end
        end
        </pre>
          </div>
        </body>
        </html>
        '

In addition, my boss uses Apple and he had a likewise problem, related with WiredTiger. However, I'm using Ubuntu and MongoDB log file doesn't show WiredTiger errors while executing tests. It seems that he fixed it with a system configuration. Apparently Rspec was trying to open all the files in order to execute the tests and the process runned out of memory or something like that. Should I have to do some similar configuration in Ubuntu?

config.ru

spec/spec_helper.rb

It seems that the route that test is trying to send a POST request to does not exist, so it shows Sinatra's 404 page rendered as HTML instead of returning a JSON payload. Did you follow the advice that shows up in that error? — gd_silva, Feb 16 '21 at 14:31
This has nothing to do with MongoDB. Make a copy of your project, remove all database references, run the tests again and update the question with findings. — D. SM, Feb 16 '21 at 15:14
@gd_silva As I mentioned, I can execute that specific test file individually than it'll pass, the problem is when running all test files together. — John David Barbosa, Feb 16 '21 at 15:21

score 0 · Answer 1 · answered Feb 18 '21 at 18:00

When test A passes in isolation, but not within the whole test suite, this usually means, that a previous test B changes the state of the test environment (e.g. doesn't clean up the database or something similar) so that test B fails.

What you need to do is to find this test B, which changes the state of the test environment and then refactor this test, so that it leaves back a clean state.

I usually do this by running test A in isolation and then go the folder structure up until test A fails and then have a look, which tests ran before.

I had a look at your spec_helper.rb and have seen, that you run your specs "in order". Once you fixed this issue I would recommend you to run them in random order. This helps you to avoid dependencies between tests. Please have a look at the Rspec documentation for details.

Add this config to your spec_helper.rb

RSpec.configure do |config|
  # other config

  config.order = "random"
  Kernel.srand config.seed
  
  # more config
end

I also recommend you, to check the response.status and the response.header['content-type'] before you parse the response.body with JSON.parse(last_response.body).

Rspec tests pass when running individually but fails when running all together

1 Answers1