2

I need to convert many TIFF images to JPEG per second. Currently I'm using libmagick++ (Q16). I'm in the process of compiling ImageMagick Q8 as I read that it may improve performance (specially because I'm only working with 8bit images).

CImg also looks like a good option and GraphicsMagick claims to be faster than ImageMagic. I haven't tested either of those yet, but I was wondering if there are any other alternatives that could be faster than using ImageMagick Q8?

I'm looking for a Linux only solution.

UPDATE width GraphicsMagick & ImageMagick Q8

Base comparison (see comment to Mark): 0.2 secs with ImageMagick Q16

I successfully compiled GraphicsMagick with Q8, but after all, it seems about 30% slower than ImageMagick (0.3 secs).

After compiling ImageMagick with Q8, there was a gain of about 25% (0.15 secs). Nice :)

UPDATE width VIPS

Thanks to Mark's post, I give it a try to VIPS. Using the 7.38 version that is found in Ubuntu Trusty repositories:

time vips copy input.tiff output.jpg[Q=95]

real    0m0.105s
user    0m0.130s
sys     0m0.038s

Very nice :)

I also tried with the 7.42 (from ppa:dhor/myway) but it seems slighlty slower:

real    0m0.134s
user    0m0.168s
sys     0m0.039s

I will try to compile VIPS from source and see if I can beat that time. Well done Mark!

UPDATE: with VIPS 8.0

Compiled from source, vips-8.0 gets practically the same performance than 7.38:

real    0m0.100s
user    0m0.137s
sys     0m0.031s

Configure command: 

./configure CC=c99 CFLAGS=-O2 --without-magick --without-OpenEXR --without-openslide --without-matio --without-cfitsio --without-libwebp --without-pangoft2 --without-zip --without-png --without-python
jcupitt
  • 10,213
  • 2
  • 23
  • 39
lepe
  • 24,677
  • 9
  • 99
  • 108
  • How big are the TIFF images - in bytes and in pixel dimensions? Are you doing any other processing on the images other than comnversion to JPEG? What happens downstream to the images? Where are they coming from? What does your code look like currently? What performance are you currently achieving? What performance do you need? – Mark Setchell May 28 '15 at 09:15
  • Thanks Mark. The images are 3288x1152. no other processing besides converting them. Images are then stored into a HDD. The raw data comes from a camera (cropped). This is how the [code looks like](http://pastebin.com/WFCnGhh0). In a testing machine (quad core) is converting one image in 0.2 seconds . I need to improve that time before moving to production (long story). – lepe May 28 '15 at 09:32
  • What are the original image sizes in bytes please? – Mark Setchell May 28 '15 at 09:51
  • around 15MB each file. – lepe May 28 '15 at 09:53
  • What JPEG quality level do you want for your output? Should the JPEGs have the same dimensions as the TIFFs (I assume yes)? Do all TIFFs have identical dimensions? – Kurt Pfeifle May 28 '15 at 12:37
  • Ha!, GraphicsMagick seems to be no longer able to sustain its claim *"I'm faster than ImageMagick!"*. I had indications of that in the past too. My overall impression is that IM has improved a *lot* in relation to its performance over the years... However, it would be nice if someone could do and publish a more complete benchmark of the two. – Kurt Pfeifle May 28 '15 at 12:41
  • @KurtPfeifle: All TIFFs have identical dimensions. Ideally, the quality level must be 95% or more. Interesting, when I use 89% it takes lot less time compared with 90%. Probably that is related to the content of the image itself. I read [here](http://blog.trackduck.com/2014/11/12/imagemagick-vs-graphicsmagick-one-better-process-images-website/) that they were/are using both (IM and GM) as both have some advantage over the other. I would also like to see a more complete benchmark. – lepe May 29 '15 at 00:42
  • Re. 89% vs. 90% -- At 90% and above, IM and GM automatically disable jpeg chroma subsample. You'll see the output files become suddenly much larger. – jcupitt Nov 18 '16 at 11:26

1 Answers1

3

I have a few thoughts...

Thought 1

If your input images are 15MB and, for argument's sake, your output images are 1MB, you are already using 80MB/s of disk bandwidth to process 5 images a second - which is already around 50% of what a sensible disk might sustain. I would do a little experiment with using a RAMdisk to see if that might help, or an SSD if you have one.

Thought 2

Try experimenting with using VIPS from the command line to convert your images. I benchmarked it like this:

# Create dummy input image with ImageMagick
convert -size 3288x1152! xc:gray +noise gaussian -depth 8 input.tif 

# Check it out
ls -lrt
-rw-r--r--@ 1 mark  staff  11372808 28 May 11:36 input.tif

identify input.tif
input.tif TIFF 3288x1152 3288x1152+0+0 8-bit sRGB 11.37MB 0.000u 0:00.000

Convert to JPEG with ImageMagick

time convert input.tif output.jpg
real    0m0.409s
user    0m0.330s
sys     0m0.046s

Convert to JPEG with VIPS

time vips copy input.tif output.jpg
real    0m0.218s
user    0m0.169s
sys     0m0.036s

Mmm, seems a good bit faster. YMMV of course.

Thought 3

Depending on the result of your test on disk speed, if your disk is not the limiting factor, consider using GNU Parallel to process more than one image at a time if you have a quad core CPU. It is pretty simple to use and I have always had excellent results with it.

For example, here I sequentially process 32 TIFF images created as above:

time for i in {0..31} ; do convert input-$i.tif output-$i.jpg; done
real    0m11.565s
user    0m10.571s
sys     0m0.862s

Now, I do exactly the same with GNU Parallel, doing 16 in parallel at a time

time parallel -j16 convert {} {.}.jpg ::: *tif
real    0m2.458s
user    0m15.773s
sys     0m1.734s

So, that's now 13 images per second, rather than 2.7 per second.

Mark Setchell
  • 191,897
  • 31
  • 273
  • 432
  • Hi, I'm the vips maintainer, `im_vips2jpeg` is the old vips7 interface. Try `time vips copy input.tif output.jpg`, the new vips8 syntax, it should be a bit quicker. – jcupitt May 28 '15 at 11:57
  • Hello, and thank you for your inputs - it is a little faster. I have edited my post accordingly. – Mark Setchell May 28 '15 at 12:18
  • @MarkSetchell: Thanks. Yes, the way I'm testing it is for example: `time convert input.tiff -quality 95% output.jpg` I compared my compiled version in C++ vs the IM binary and it was pretty much the same times. – lepe May 29 '15 at 00:48
  • About "GNU Parallel", I'm using boots::thread in my aplication. Technically should be similar or not? Perhaps I should mention that in production, we will use a Xeon E3-1271V3 with 4 cores (8 threads) to process 4 cameras at the same time storing at 4 HDD. I'm using a RAMdisk to store the TIFF files (placed at /tmp/ as IM also uses it to place temporally data) to convert the them to JPG before they hit the HDD bandwidth. All my tests and numbers seems ok, however I haven't test that processor yet, that is why I wan't to be sure to be able to convert them as fast as possible. I will test VIPS – lepe May 29 '15 at 00:58
  • MarkSetchell: I think the goal was achieved. Your recommendation was 2 times faster, which is more than I expected. Thanks! @user894763: As this is about C++ and you are a vips maintainer, do you know which parameters do I need to set in [vips_copy](http://www.vips.ecs.soton.ac.uk/supported/7.32/doc/html/libvips/libvips-conversion.html#vips-copy) to convert from TIFF to JPG? I coudn't find an example. Any help is appreciated. – lepe May 30 '15 at 01:59
  • 1
    @lepe I made you a sample program https://gist.github.com/jcupitt/279ac81956b864e03e6c VIPS C++ docs here http://www.vips.ecs.soton.ac.uk/supported/current/doc/html/libvips/using-from-cpp.html open an issue on the vips tracker if you have problems – jcupitt May 30 '15 at 21:01
  • @user894763: Thank you so much! – lepe Jun 01 '15 at 01:26