What is the formula for setting WEB_CONCURRENCY on heroku?

Question

I am frequently getting this error Memory quota exceeded for my ruby on rails application hosted on heroku:

2014-12-29 11:09:37.876 355 <45>1 2014-12-29T11:09:37.210533+00:00 heroku worker.1 - - source=worker.1 dyno=heroku.22144543.00c11b4d-8ec6-46d9-addf-f03163e10f0c sample#memory_total=2899.25MB sample#memory_rss=1023.73MB sample#memory_cache=0.00MB sample#memory_swap=1875.52MB sample#memory_pgpgin=2603236pages sample#memory_pgpgout=2341160pages
2014-12-29 11:09:37.876 132 <4
2014-12-29 11:09:37.876 132 <455>1 2014-12-29T11:09:37.210533+00:00 heroku worker.1 - - Process running mem=2899M(283.1%)>1 2014-12-29T11:09:37.210533+00:00 heroku worker.1 - - Error R14 (Memory quota exceeded)

I read the blog of which suggests the solution i.e. set WEB_CONCURRENCY config variable but it is not clear to me.

My current configuration on heroku are:

config/unicorn.rb

worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)
timeout 29
preload_app true

Edited:

Worker Code

class PhotoWorker
  include Sidekiq::Worker
  sidekiq_options queue: "high"
  # sidekiq_options retry: false

  def perform(params)    
    begin
      puts "Start Woker for #{params.inspect}"
      site = Site.find params['site_id']      
      if params["id"].present?
        photo = Photo.find(params["id"])
        if params['key'].present?
          photo.key = params['key']
          photo.remote_image_url = photo.image.direct_fog_url(:with_path => true)
          photo.image_name = params['key'].split('/').last
          photo.save(validate: false)
          puts photo.inspect
        end     
      else     
        #photo = site.photos.build 
        #photo.hotel_detail_id = site.hotel_detail.id
        #photo.album_id = site.hotel_detail.album.id
        #photo.save(validate: false)
        photo.key = params['key']
        photo.remote_image_url = photo.image.direct_fog_url(:with_path => true)
        #photo.image = params['key']
        #photo.remote_image_url = params['key']
        photo.image_name = params['key'].split('/').last
        puts photo.inspect
        photo.save(validate: false)
      end

      if params["id"].present? && params["crop_x"].present? 
        photo.crop_x = params["crop_x"]
        photo.crop_y = params["crop_y"]
        photo.crop_w = params["crop_w"]
        photo.crop_h = params["crop_h"]
        photo.save(validate: false)
      end


      if params["id"].present? 
        if params['key'].present?
          s3 = AWS::S3.new.buckets[ENV["FOG_DIRECTORY"]]
          s3.objects[params['key']].delete
        end
      else
        AmazonFile.new(params['key']).delete
      end

      puts "Deleted temp file: #{params['key']}" if params['key'].present?
      puts "Photo(#{photo.id}) saved successfully. URL: #{params['key']}"
    rescue Exception => exc
      #puts "Photo not saved. URL: #{params['key']}" 
      puts "Error:  #{exc.message}"
    end      
  end

end

Apart from sidekiq i am also using carrierwave_backgrounder gem.

I am using Kraken for image compression

Photo Uploader

class PhotoUploader < CarrierWave::Uploader::Base 
  #include ::CarrierWave::Backgrounder::Delay
  include CarrierWaveDirect::Uploader
  include CarrierWave::MiniMagick
  include CarrierWave::MimeTypes
  process :set_content_type
  process :crop  
  storage :fog
  after :store, :kraken

  version :admin do
    process :resize_to_fit => [200,200]
  end

  #Leisure theme
  version :leisure_358x243, :if => :leisure_theme? do
    process :resize_to_fill => [358,243]
  end
  version :leisure_900x500, :if => :leisure_theme? do
    process :resize_to_fill => [900,500]
  end
 version :leisure_350x147, :if => :leisure_theme? do
    process :resize_to_fill => [350,147]
  end
  version :leisure_1100x344, :if => :leisure_theme? do
    process :resize_to_fill => [1100,344]
  end

  #Business theme
  version :business_360x160, :if => :business_theme? do
    process :resize_to_fill => [360,160]
  end
  version :business_1100x315, :if => :business_theme? do
    process :resize_to_fill => [1100,315]
  end
  version :business_1140x530, :if => :business_theme? do
    process :resize_to_fill => [1140,530]
  end
  version :business_1100x355, :if => :business_theme? do
    process :resize_to_fill => [1100,355]
  end

  #Commthree theme
  version :commthree_550x300, :if => :commthree_theme? do
    process :resize_to_fill => [550,300]
  end
  version :commthree_319x183, :if => :commthree_theme? do
    process :resize_to_fill => [319,183]
  end
  version :commthree_1920x700, :if => :commthree_theme? do
    process :resize_to_fill => [1920,700]
  end

  #All theme
  version :all_360x188 do
    process :resize_to_fill => [360,188]
  end
  version :all_1100x401 do
    process :resize_to_fill => [1100,401]
  end
  version :all_140x88 do    
    process :resize_to_fill => [140,88]
  end

  def kraken(file)
    if version_name.to_s == ""      
      storepath = store_dir + "/" + filename
    else
      fname = filename.split('.')
      #originalfile = "http://s3.amazonaws.com/" + ENV["FOG_DIRECTORY"] + "/" + store_dir  + "/" + fname[0] + "_" + version_name.to_s + "." + fname.last
      storepath = store_dir + "/" + fname[0] + "_" + version_name.to_s + "." + fname.last
    end

    originalfile = "http://s3.amazonaws.com/" + ENV["FOG_DIRECTORY"] + "/" + store_dir + "/" + filename

    kraken = Kraken::API.new(
        :api_key => ENV['KRAKEN_API'],
        :api_secret => ENV['KRAKEN_SECRET']
    )

    params = {}


    Rails.logger.info('Kraken About to Process: ' + originalfile);
    p "version #{version_name}"
    case version_name.to_s
      when "admin"
        params = {
                'lossy' => true,
                'resize' => {
                     'width' => 200,
                     'height' => 200,
                    'strategy' => "auto",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }

      when "leisure_358x243"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 358,
                    'height' => 243,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }
      when "leisure_900x500"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 900,
                    'height' => 500,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }           
      when "leisure_350x147"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 350,
                    'height' => 147,
                    'strategy' => "exact",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }
      when "leisure_1100x344"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 1100,
                    'height' => 344,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }

      when "business_360x160"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 360,
                    'height' => 160,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }                
      when "business_1100x315"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 1100,
                    'height' => 315,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }                
      when "business_1140x530"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 1140,
                    'height' => 530,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }
      when "business_1100x355"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 1100,
                    'height' => 355,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }

      when "commthree_550x300"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 550,
                    'height' => 300,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }
      when "commthree_319x183"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 319,
                    'height' => 183,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }
      when "commthree_1920x700"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 1920,
                    'height' => 700,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }

      when "all_360x188"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 360,
                    'height' => 188,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }
      when "all_1100x401"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 1100,
                    'height' => 401,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }
      when "all_140x88"
        params = {
                'lossy' => true,
                'resize' => {
                    'width' => 140,
                    'height' => 88,
                    'strategy' => "fit",
                },
                's3_store' => {
                              'key' => ENV['AWS_ACCESS_KEY_ID'],
                              'secret' => ENV["AWS_SECRET_ACCESS_KEY"],
                              'bucket' => ENV["FOG_DIRECTORY"],
                              'acl' => 'public_read',
                              'path' => storepath
                              },
                  }
    end


    #Store the file online

    if !version_name.blank?

      Rails.logger.info('UPLOADING TO: ' + store_path);

      data = kraken.url(originalfile,params)

      if data.success
          Rails.logger.info('KRAKEN: Success! Optimized image URL: ' + data.kraked_url)
      else
          puts 
          Rails.logger.info('KRAKEN: Fail. Error message: ' + data.message)
      end

    end    
  end

  # Add a white list of extensions which are allowed to be uploaded.
  # For images you might use something like this:
  def extension_white_list
    %w(jpg jpeg gif png)
  end

  # def cache_dir    
  #   "#{Rails.root}/tmp/uploads"
  # end

  def store_dir
    if model.id.present?
      "photos/#{model.id}"
    else
      "photos"
    end
  end

  #file name is missing extension!!!
  def filename
    original_filename if original_filename.present?
  end

  def crop
     puts "i am uploader"
    if model.crop_x.present?
      manipulate! do |img|
        puts "i am cropping"
        cx = model.crop_x.to_i
        cy = model.crop_y.to_i
        cw = model.crop_w.to_i
        ch = model.crop_h.to_i        
        img.crop"#{cw}x#{ch}+#{cx}+#{cy}"
        #img
      end
    end
  end

  def business_theme? (image)
    p "model:#{model.id}"
    (model.hotel_detail.site.theme.layout == "x1")if model.id.present?
  end

  def leisure_theme? (image)
    p "model:#{model.id}"
    (model.hotel_detail.site.theme.layout == "x2")if model.id.present?
  end

  def commthree_theme? (image)
    p "model:#{model.id}"
    (model.hotel_detail.site.theme.layout == "x3") if model.id.present?
  end

  protected
  def secure_token(length=16)
    var = :"@#{mounted_as}_secure_token"
    model.instance_variable_get(var) or model.instance_variable_set(var, SecureRandom.hex(length/2))
  end

end

Please suggest me how to calculate the correct value of WEB_CONCURRENCY.

Answer 1

It looks like you are using both minimagick and kraken to do image manipulation. If you fully migrate to Kraken and let them handle all of your image manipulation you shouldn't have a problem with memory on your workers.

Remove all of your versions and see if that improves performance.

version :leisure_358x243, :if => :leisure_theme? do
  process :resize_to_fill => [358,243]
end

If you want to stick with processing images on your workers there are more efficient libraries than carrierwave-minimagick. Take a look at carrierwave-vips.

Answer 2

There is no "magic formula" - as per the Heroku docs for Deploying Rails Applications on Unicorn - Caveats:

Each forked OS process consumes additional memory. This limits how many processes you can run in a single dyno. With a typical Rails memory footprint, you can expect to run 2-4 Unicorn worker processes. Your application may allow for more or less processes depending on your specific memory footprint, and we recommend specifying this number in an config var to allow for faster application tuning.

Additionally, specific versions of Ruby run better on Heroku than others, due to memory leaks. I strongly suggest Ruby 2.0.0 for the best memory maintenance with the New Relic gem, as the latest New Relic gem has evidence of a memory leak. I personally had issues with the New Relic gem on Ruby 2.1.1, 2.1.3, and 2.1.4. I've since downgraded to 2.0.0, which cut my memory usage in half.

However, if you're concerned about performance of Ruby 2.0.0, you might want to check out an alternative to New Relic, at least until New Relic patches the memory leak in their latest gem.

Answer 3

Just to react about CDub's answer. I had a couple of memory issues with the last Ruby version. Yes, New Relic seems to use a lot of memory but it's not the reason of those Memory quota exceeded errors. After reading a lot of posts talking about that problem, it seems that the real cause come from the Ruby GC, which is causing a lot of server swapping since version 2.1.x .

To fix that I just set RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR to a value around 1.25. You can play with that setting to find the best value for your environment.

FYI my app is running with Ruby 2.1.5 on Heroku Cedar 14

Hope it helps :)