0

So I have a PORO service where I get one image at time and store this original image, after that I scheduled a sidekiq job where I convert this image to webp format with three different dimensions. But I noticed that sidekiq consumes ~200MB of memory at the beginning and when it starts processing 4MB image(.jpeg) it quickly grows to ~350MB. And if the user sends 8 consecutive requests with the total image size of ~18mb the job may take up to 800MB and this memory is not freed after completion. Therefore, any further requests only increase the memory of the job. I'm running docker on linux machine, also i'm using plain ActiveStorage and image_processing gem with libvips image processor. Anyone having the same problem with this or know how to decrease the memory?

Here is the code of job:

class Api::V1::Ads::Images::ResizeAndUploadJob < Api::V1::ApplicationJob
  sidekiq_options queue: 'high'

  def perform(blob_id)
    @blob = ActiveStorage::Blob.find_by(id: blob_id)

    return if @blob.nil?

    @blob.filename = "#{image_filename}_x1200.webp"
    @blob.variant(format: :webp, resize_to_limit: [nil, 1200]).process

    @blob.filename = "#{image_filename}_x560.webp"
    @blob.variant(format: :webp, resize_to_limit: [nil, 560]).process

    @blob.filename = "#{image_filename}_x130.webp"
    @blob.variant(format: :webp, resize_to_limit: [nil, 130]).process
  end

  private

  def image_filename
    @image_filename ||= @blob.filename.base.split('_ORIGINAL').first
  end
end
CR7
  • 1,056
  • 1
  • 8
  • 18
  • Yeap noticed the bloat years ago. But never been able to track down the issue. Ended up using an external to version images. – Maxence Nov 25 '22 at 11:51
  • @Maxence you mean that you start using external image processing system? – CR7 Nov 25 '22 at 12:12
  • Yep. A homemade service on bare metal as my Heroku Dyno couldn't cope with the Ram increase. Yet I would be interested to know if there is a solution to this as VIPS doesn't really eat much RAM. – Maxence Nov 25 '22 at 15:16

1 Answers1

2

The libwebp WebPDecode() function is one shot rather than incremental, meaning you have to load the whole of the compressed input file into memory, allocate enough ram for the whole of the decompressed pixel array, then decompress the entire thing in a single call into libwebp.

This means that large webp images will need a lot of memory to process. Although this memory is freed again after the resize is done, heap fragmentation means that it can take a while for overall memory use to stabilize, and the level it settles at might be higher than you'd expect.

Workarounds:

  1. A malloc that tries to avoid fragmentation, like jemalloc, can help a lot.

  2. Don't use webp for large images if you can help it (not always possible, of course).

  3. The libvips operation cache can mean memory is kept for longer than you'd expect. You can try turning the cache size down with cache_set_max().

libwebp now has API to do incremental decoding and encoding, but no one's got around to adding support to libvips yet. There's an open issue on this:

https://github.com/libvips/libvips/issues/3077

jcupitt
  • 10,213
  • 2
  • 23
  • 39
  • how do you think, if I’d rather convert every image to .jpeg will it be more efficient in terms of memory usage? My initial problem was to save images in efficient way to have faster page loading. The image size could be up to 5MB, but in future it could be up to 10MB. Do you think that .jpeg will be enough, or I don’t need even to convert them, just resize should be fine? – CR7 Nov 27 '22 at 19:32
  • I noticed that even without your recommendations but whith concurrency: 2 set in sidekiq.yml the memory goes to 800mb and then begin decreasing to 500 - 600. jemalloc didn't help too – CR7 Dec 06 '22 at 21:50
  • I don't think webp is worth using over jpeg for most people. It's annoying to set up, and the image size savings are minimal compared to mozjpeg. – jcupitt Dec 07 '22 at 09:49