How to deal with memory leaks in RMagick in Ruby?

Question

Im developing web-application with Merb and im looking for some safe and stable image processing library. I used to work with Imagick in php, then moved to ruby and start using RMagick. But there is a problem. Long running scripts causing memory leaks. There are couple solution exists, but I don't know which one is the most stable. So, what do you think?

Right now, my app uses internal API that i wrote to process images, in PHP. Its running on separate server along with other applications, so its not a big problem. But i think its not a good architecture.

Anyway, i`ll consider any practical tips.

Answer 1

I too have encountered this issue - the solution is to force garbage collection.

When you have reassigned the image variable to a new image simply use GC.start to ensure the old reference is released from memory.

On later versions of RMagick, I also believe you can also call destroy! on the image when you have finished processing it.

A combination of the two would probably ensure you are covered, but im not sure of the real life impact on performance (I would assume it is negligible i most cases).

Alternatively, you could use mini-magick which is a wrapper for the ImageMagick commandline client.

Answer 2

When using RMagick it's important to remember to destroy the image once you are done, otherwise you will fill up the /tmp dir when working with large sets of images. For example you must call destroy!

require 'RMagick'

Dir.foreach('/home/tiffs/') do |file|
    next if file == '.' or file == '..'
        image = Magick::Image.read(file).first
        image.format = "PNG"
        image.write("/home/png/#{File.basename(file, '.*')}.png")
        image.destroy!
end

Answer 3

Actually, it isn't really a Ruby specific problem, other Interpreters share that as well. The concrete problem is that the GC of Ruby only sees memory that was allocated by Ruby itself, and not by external libraries (with the notable exception of the library using Rubys memory management facilities). So, a ImageMagick-Object in Ruby memory space is really small, but the image in the space managed by ImageMagick is large. So, this is not a leak per se, but it behaves like one. Rubys Garbage Collector never kicks in if your Process stays under a certain limit (8MB is standard). As ImageMagick never creates large objects in Ruby space, it probably never kicks in. So, either you use the proposed method of spawning a new process or using exec. Another rather nifty one is to have an image processing service in the backend that forks for every task. Another one would be to have some kind of monitoring in place that kickstarts the GC every once in a while.

There is another Library called MagickWand by Timothy Paul Hunter (the author of RMagick) that tries to address these issues and create a nicer API. It's in alpha and requires a rather new release of ImageMagick, though.

Answer 4

Now you can tell ImageMagick which memory space should be used. I think RMAGICK_ENABLE_MANAGED_MEMORY = true and GC.start is what you need.

MANAGED_MEMORY

If true, RMagick is using Ruby managed memory for all allocations. If false, 
RMagick allocates memory for objects directly from the operating system. You can 
enable RMagick to use Ruby managed memory (when built with ImageMagick 6.4.0-11 
and later) by setting
RMAGICK_ENABLE_MANAGED_MEMORY = true
before requiring RMagick.

https://rmagick.github.io/constants.html

However, image.destroy! itself is enough to stabilize the memory consumption.

Answer 5

This is not due to ImageMagick; it's due to Ruby itself, and it's a well known problem. My suggestion is to split your program into two parts: a long-running part that allocates little memory and just deals with the control of the system, and a separate program that actually does the processing work. The long-running control process should do just enough to find some work for a child process that it spawns, and the child should do all of the processing for that particular work item.

Another option would be to leave the two combined, but after a work unit is complete, use exec to replace your process with a freshly started version of the same program, which would search for another work item, process it, and exec itself again.

This is assuming that the work items are fairly large, which they almost certainly are if you're using ImageMagick. If they're not, you'll find that the overhead of spawning a new process and having the Ruby interpreter re-parse your entire program starts to get a little too large. You can deal with this by having your program do more work units (say, ten or a hundred) before re-executing itself.