37

I have been getting the (apparently common) Error R14, not sure when it started but noticed after installing the Papertrail add-on. I also added oink after seeing others mentioning it. I have tried everything others have tried in other SO questions but I can't seem to find the problem / memory leak.

Error R14 (Memory quota exceeded)
heroku/web.1: Process running mem=587M(114.7%)

When I see this, to temporarily fix this, I run heroku restart which works for a while but I want to fix this problem permanently.

From what I can see in Papertrail, the app just keeps loading the same page (home page) over and over again (and its not user traffic):

app/web.1: Completed 200 OK in 436ms (Views: 45.5ms | ActiveRecord: 386.2ms) 
app/web.1: May 25 18:14:52 5d2105e1-d799-4496-a2af-3785e78998db rails[9]: Oink Action: static_pages#home
app/web.1: May 25 18:14:52 5d2105e1-d799-4496-a2af-3785e78998db rails[9]: Memory usage: 378860 | PID: 19
app/web.1: May 25 18:14:52 5d2105e1-d799-4496-a2af-3785e78998db rails[9]: Oink Log Entry Complete 
heroku/web.1: source=web.1 dyno=heroku.25566769.9d9a3da0-db4c-4b51-bc53-b69be9e43cb7 sample#memory_total=209.86MB sample#memory_rss=209.83MB sample#memory_cache=0.02MB sample#memory_swap=0.00MB sample#memory_pgpgin=59278pages sample#memory_pgpgout=5555pages

and after 2 minutes:

heroku/web.1: source=web.1 dyno=heroku.25566769.9d9a3da0-db4c-4b51-bc53-b69be9e43cb7 sample#memory_total=293.73MB sample#memory_rss=291.94MB sample#memory_cache=0.02MB sample#memory_swap=1.77MB sample#memory_pgpgin=80890pages sample#memory_pgpgout=6147pages `

I also enabled heroku labs:enable log-runtime-metrics from instructions from another question here and from Heroku here

I have also added the New Relic add-on and this is what I see in the Monitoring > Instances tab:

For the Last 7 Days: last 7 days

For the Last 24 Hours: last 24 hours

I am confused as you can see after 5/24 the memory usage went down and has since stayed down, but in the logs it shows memory being used mem=587M(114.7%) and in the last 24 hours an average of 150M, so I don't understand what is happening. I hope someone can help. Thanks.

Update 1: Added Gemfile

#Gemfile
source 'https://rubygems.org'
ruby '2.1.1'
gem 'rails', '4.1.1'

gem 'puma', '~> 2.8.2'
gem 'turbolinks', '~> 2.2.2'
gem 'pg', '~> 0.17.1'
gem 'rack-attack', '~> 4.0.1'
gem 'memcachier'
gem 'dalli', '~> 2.7.1'
gem 'oink'
gem 'fog', '~> 1.22.0'
gem 'activeadmin', github: 'gregbell/active_admin'
gem 'dynamic_sitemaps', '~> 2.0.0'
gem 'whenever', :require => false
gem 'cancancan', '~> 1.8.0'
gem 'mini_magick', '~> 3.7.0'
gem 'carrierwave', '~> 0.10.0'
gem 'devise', '~> 3.2.4'
gem 'rolify', '~> 3.4.0'
gem 'simple_form', '~> 3.0.2'
gem 'cocoon', '~> 1.2.6'
gem 'friendly_id', '~> 5.0.3'
gem 'nokogiri', '~> 1.6.2.1'
gem 'kaminari'
gem 'impressionist', '~> 1.5.1'
gem 'validate_url'
gem 'searchkick', '~> 0.7.5'
gem 'meta-tags', '~> 2.0.0'
gem 'newrelic_rpm'


group :assets do
  gem 'sass-rails', '~> 4.0.3'
  gem 'bootstrap-sass', '~> 3.1.1.1'
  gem 'uglifier', '~> 2.5.0'
  gem 'coffee-rails', '~> 4.0.1'
  gem 'asset_sync'
  #gem 'jquery-turbolinks'
  gem 'jquery-rails'
  gem 'jbuilder', '~> 2.0.7'
end

group :production do
  gem 'rails_12factor'
end

Update 2: Changed server from Puma to Unicorn

It was pretty simple, following the directions here. And this is what it looks like now, which seems good, but I suspect it was because the application was restarted: last30mins

# after heroku restart
heroku/web.1: source=web.1 dyno=heroku.25566769.3770e52a-ddf3-485a-a88a-ac4fd54ab640 sample#memory_total=123.62MB

# after 4 minutes
heroku/web.1: source=web.1 dyno=heroku.25566769.3770e52a-ddf3-485a-a88a-ac4fd54ab640 sample#memory_total=276.48MB

Update 3: Lowered the amount of Unicorn workers to 2

After doing that, this is the average memory usage per dyno/instance for the past 6 hours according to New Relic: lasy6hoursafter

And the output in PaperTrain (and also tried LogEntries) was this:

app/web.1: May 26 19:54:08 21ae35ee-5656-4254-9f12-5dc6bb59efa1 rails[6]: Memory usage: 426076 | PID: 6
app/web.1: May 26 19:54:08 21ae35ee-5656-4254-9f12-5dc6bb59efa1 rails[6]: Oink Log Entry Complete
heroku/web.1: source=web.1 dyno=heroku.25566769.21ae35ee-5656-4254-9f12-5dc6bb59efa1 sample#load_avg_1m=0.00 sample#load_avg_5m=0.00 sample#load_avg_15m=0.00
heroku/web.1: source=web.1 dyno=heroku.25566769.21ae35ee-5656-4254-9f12-5dc6bb59efa1 sample#memory_total=431.99MB sample#memory_rss=406.85MB sample#memory_cache=0.10MB sample#memory_swap=25.04MB sample#memory_pgpgin=198612pages sample#memory_pgpgout=94432pages

I checked the logs, and after 1 hour I was getting the R14 error and now it has been been going up very little, almost holding steady at 432MB and have not gotten the error anymore. So this seems to have fixed the problem! I will update over time if this changes.

thecrentist
  • 1,241
  • 2
  • 19
  • 28
  • 1
    What ruby version are you using? Post your Gemfile as well. – Mike Szyndel May 25 '14 at 21:07
  • 1
    @MichaelSzyndel I am using ruby 2.1.1 and added gemfile – thecrentist May 25 '14 at 21:35
  • 2
    AFAIK Puma is not really designed for MRI, not saying that this is the reason but can you try Unicorn? (Using Puma doesn't give you anything anyway). Quote from the website: "Puma is designed to be used on a Ruby implementation which provides true parallelism, such as Rubinius and JRuby." – Mike Szyndel May 25 '14 at 22:08
  • 1
    @MichaelSzyndel I changed the server like you suggested and as you can see in "Update 2" it helped somehow. – thecrentist May 25 '14 at 22:59

5 Answers5

17

If Papertrail is causing a problem, try a different add on. I've been using LogEntries without much of a problem. https://addons.heroku.com/#logging

Also try to lower your Unicorn worker processes so it uses lower total memory. Instead of the default of 3 (per box/dyno), try 2.

https://devcenter.heroku.com/articles/rails-unicorn#unicorn-worker-processes

You can also run a memory profiler on your app:

http://timetobleed.com/memprof-a-ruby-level-memory-profiler/

https://www.ruby-toolbox.com/search?utf8=%E2%9C%93&q=profile

Chloe
  • 25,162
  • 40
  • 190
  • 357
  • 1
    I used Papertrail on heroku with Unicorn for a long time and had no problem with it. Strange anyway. – Mike Szyndel May 26 '14 at 07:08
  • 1
    What really worked for me was lowering the worker processes to 2 – Abs Sep 29 '14 at 23:23
  • 1
    How do you "lowering the worker processes"? Does that mean setting the `WEB_CONCURRENCY` environment variable? – sguha Feb 05 '15 at 02:07
  • 1
    @sguha just check `config/unicorn.rb`, mine defaulted with `worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)` meaning that it looks for the `WEB_CONCURRENCY` env variable, then defaults to 3 if it doesn't find it. – Kyletns Apr 22 '15 at 00:12
  • 1
    Papertrail doesn't interact with your app nor add any load to it. It interacts with your logs via what Heroku calls a "log drain". – xentek Dec 10 '15 at 02:11
13

Digital Ocean wrote a great article about this.

Killing Unicorns can work in certain situations... however some people may have ethical concerns with this approach.

Sky
  • 4,327
  • 4
  • 26
  • 40
8

If you're using Rails 4 with Ruby 2.1, I bet the only change you need to make is downgrading to Ruby 2.0 back.

I spent 2 days to figure it out and I came up with this idea with Heroku support's suggestions. I think that's a bug or something like that with Ruby 2.1 or Rails 4.

Update by April 2015:

At the time, downgrading did work as a solution. But well, it's really not a recommended way of solving problems... Some people say upgrading to Ruby 2.2.0 with Rails 4.2 significantly decreases memory consumption. I would give it a try!

scaryguy
  • 7,720
  • 3
  • 36
  • 52
  • 2
    Downgrading to Ruby 2.0 fixed the memory quota errors, but I found my app ran noticeably slower. I upgraded to Ruby 2.1.3 and the R14 errors have not returned. You can turn on memory stats in the logs using log-runtime-metrics. Based on those log entries, my average memory usage dropped by about 100 megs over Ruby 2.1.2. – SteveO7 Oct 10 '14 at 16:03
  • 1
    I'm still getting R14 errors when upgrading to 2.1.3. Had to downgrade to 2.0.0. Using Heroku and Unicorn. – CoderDave Oct 22 '14 at 21:40
  • 1
    Downgrading to Ruby 2.0 seems to be working for me. Had the problem with 2.1.3 and 2.1.5. – Tim Scott Jan 02 '15 at 05:37
  • 1
    I've had this problem since 18 November 2014, but I upgraded from Ruby 2.0 to Ruby 2.1 (version 2.1.3 to be precise) on 9 September 2014. Ruby 2.1 is a lot faster than 2.0 was though. I see R14 messages occurring after a restart after a couple of hours of typical usage on my app, am running two unicorns (I used to run three comfortably on Ruby 2.0) and am still on Rails 4.0. Are people also seeing this with Rails 4.1 or 4.2? – Graham Ashton Jan 08 '15 at 09:29
  • 6
    For what it's worth, I just upgraded to Ruby 2.2.0 and my Heroku memory consumption dropped significantly from Ruby 2.1.3. I'm using Rails 4.2.0 – SteveO7 Jan 13 '15 at 17:28
  • Puma running on ruby 2.2.3 with rails 4.0.13, problem persists... I'll try to upgrade to 4.2 – Rafael Oliveira Sep 10 '15 at 19:03
  • kudos for you. Upgrading from 2.1.5 to 2.3.1 solved my worker memory consumption problem. – Paniko0 Oct 20 '16 at 23:58
0

Ruby 2.1.1 had a bug in Garbage Collector well described by Sam Saffron:

http://samsaffron.com/archive/2014/04/08/ruby-2-1-garbage-collection-ready-for-production

It was causing kind of a memory leak. The solution is to upgrade Ruby. Current Heroku supported versions are 2.1.7 and 2.2.3.

Victor BV
  • 1,051
  • 13
  • 9
0

I had a similar issue. I am using Ruby 2.3.1 and Rails 4.2.6 , Heroku with Unicorn. Following two configuration variables were not set in my app before. Setting them from the terminal with the following command resolved the issue for me.

heroku config:set sensible_defaults=disabled
heroku config:set WEB_CONCURRENCY=1
techdreams
  • 5,371
  • 7
  • 42
  • 63