11

I built a pretty small Rails 5.1.4 (Ruby 2.3.1) application. Once I've deployed it to production, I'm getting this particular error from time to time:

RuntimeError: can't add a new key into hash during iteration

Pointing here:

# rack/request.rb, line 67
def set_header(name, v)
  @env[name] = v
end

I understand, this error happens when you're attempting to add new key to the hash while iterating over that hash. Since @env is a hash, it makes sense. But:

  1. in a stacktrace I found nothing related to iterations over @env, it's a dead simple chain of app.call(env) calls.
  2. this error occurs not always, but just once per hour or two, so this is also super weird to me
  3. I can't reproduce it locally: I've tried to load server with multiple request hits, assuming this might be thread-safety issue, but locally it works like a charm...

Full stacktrace only consists of rack middlewares can be found here: https://gist.github.com/Nattfodd/e513122400b4115a653ea38d69917a9a

Gemfile.lock: https://gist.github.com/Nattfodd/a9015e9204544302bf3959cec466b715

Server is running with puma, config is very simple: just amount of threads and workers:

threads 0, 5
workers 5

My current ideas are:

  • one of monitoring gems has a bug (sentry-raven, new_relic)
  • concurrent-ruby has a bug (I read about one, but it was fixed in 1.0.2, and actual version I'm using for Puma is 1.0.5)
  • something super stupid, I missed, but I have no idea where to look, since the controller's action contains 3 lines of code and application config is mostly default...
  • this is something config-related, since backtrace does not contain the controller at all...
nattfodd
  • 1,790
  • 1
  • 17
  • 35
  • 1
    did you ever figure this out? im having the same problem – Serge Pedroza Jan 09 '18 at 19:17
  • 2
    Yeah, the reason is `newrelic_rpm` latest version(s). Still dunno what exactly causes this error in newrelic gem. – nattfodd Jan 09 '18 at 19:42
  • Thanks for the response. Any suggestions in terms of what version of new relic to use that is stable and wont cause this problem? – Serge Pedroza Jan 09 '18 at 22:15
  • 1
    4.6.0 is broken for sure, can't say if 4.7 or 4.5 is okay - I didn't test it yet – nattfodd Jan 10 '18 at 10:31
  • I'm still seeing this w/ newrelic_rpm 5.0.0.342 although I can't say for certain it's the cause – swrobel Apr 26 '18 at 15:47
  • 2
    A similar bug has been intermittently affecting my production application for 2+ years, with versions of `newrelic_rpm` ranging from v3.16.0.318 through v4.8.0.341, on both unicorn and puma. One difference is that the top of my stacktrace is in an `aws-sdk` client connection-pool handler ([issue](https://github.com/aws/aws-sdk-ruby/issues/1331)), but since that hasn't turned up anything obvious I'm increasingly suspecting New Relic instrumentation as the culprit. The other thing my stacktrace has in common is that it also includes both sinatra + activerecord- maybe that's also relevant? – wjordan Oct 28 '18 at 02:24

1 Answers1

-1

can you can paste the full stack-trace? My assumption is set_header is getting called from method which iterating env.

Ravi Prakash
  • 123
  • 1
  • 7
  • 1
    the link to the full stacktrace is in my question - https://gist.github.com/Nattfodd/e513122400b4115a653ea38d69917a9a – nattfodd Dec 12 '17 at 09:40