ImageMagick single convert command performance

Question

I have a few thousand images to be processed so each millisecond counts. Each image is ~2-3Mb in size.

Source file fed to the converter: image.jpg

Files to be generated out of the source:

orig_image.jpg      // original image
1024x768_image.jpg  // large image
250x250_image.jpg   // thumbnail 1
174x174_image.jpg   // thumbnail 2

While browsing different topics on imagemagick convert performance I got a feeling that a single command should be way faster than individual converts for each image size. Also a memory utilization was mentioned as a performance boost. (ImageMagick batch resizing performance)

Multiple command conversion (each command run via php's exec() in a loop):

convert "image.jpg" \
    -coalesce -resize "1024x768>" +repage "1024x768_image.jpg"

convert "1024x768_image.jpg" \
    -coalesce \
    -resize "250x250>" \
    +repage \
    -gravity center \
    -extent "250x250" "250x250_image.jpg"

convert "1024x768_image.jpg" \
    -coalesce \
    -resize "174x174>" \
    +repage \
    -gravity center \
    -extent "174x174" "174x174_image.jpg"

mv image.jpg orig_image.jpg

Single command conversion incorporating ImageMagicks mpr:

convert "image.jpg" -quality 85 -colorspace rgb -coalesce \
    -resize "1024x768>" \'
    -write "1024x768_image.jpg" \
    -write mpr:myoriginal +delete \
    mpr:myoriginal -coalesce \
    -resize "250x250>" \
    -gravity center \
    -extent "250x250" \
    -write "250x250_image.jpg" +delete \
    mpr:myoriginal -coalesce \'
    -resize "174x174>" \
    -gravity center \
    -extent "174x174" \
    -write "174x174_image.jpg"

After performance testing the results are somewhat unexpected. Single command convert in a loop finishes in 62 seconds while multiple command conversion executes in just 16 seconds?

# convert -version
Version: ImageMagick 7.0.2-1 Q8 i686 2017-02-03 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2016 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC HDRI OpenMP
Delegates (built-in): bzlib freetype jng jpeg lzma png tiff wmf xml zlib

Also installed libjpeg-turbo jpg processing library but I cannot tell (don't know how to check) if ImageMagic is using it or the old libjpeg.

Any ideas how to speed up image converting process?

Edit: Don't know how to format it properly here on stackoverflow, but I just noticed that single line command had an argument "-colorspace rgb" and multiple line commands did not which actually results in such strange results where multiple commands are processed faster.

Removed the "-colorspace rgb" argument after which the MPR convert version works the best and gave additional boost in performance.

To sum it all up I ended up using this command:

// MPR
convert "orig_image.jpg" -quality 80 -coalesce \
    -resize "1024x768>" \
    -write 1024x768_image.jpg \
    -write mpr:myoriginal +delete \
    mpr:myoriginal -resize "250x250>" \
    +repage -gravity center -extent "250x250" \
    -write "250x250_image.jpg" \
    -write mpr:myoriginal +delete \
    mpr:myoriginal -coalesce -resize "174x174>" \
    +repage -gravity center -extent "174x174" \
    -write "174x174_image.jpg"

strace convert... shows that it actually picks up library files from jpeg-turbo. Image in this case is not relevant, size around 2mb and larger than 1024x768 — Didzis, Feb 06 '17 at 09:20
I would add `-define jpeg:size=2048x1536` as the first argument, it should give you an easy x2 speedup, assuming your originals are larger than 4096x3072. — jcupitt, Feb 06 '17 at 10:57

score 4 · Answer 1 · answered Feb 03 '17 at 13:00

You're not using jpeg shrink-on-load, that'll give an easy speedup.

The jpeg library has a neat feature where it'll let you decompress at full resolution, at 1/2, 1/4 or 1/8th. 1/8th resolution is especially quick because of the way jpg works internally.

To exploit this in convert you need to hint to the jpeg loader that you need an image of a particular size. To avoid aliasing you should ask for an image at least 200% larger than your target size.

On this machine, I see:

$ vipsheader image.jpg 
image.jpg: 5112x3470 uchar, 3 bands, srgb, jpegload
$ time convert image.jpg -resize 1024x768 1024x768_image.jpg
real    0m0.405s
user    0m1.896s
sys 0m0.068s

If I set the shrink-on-load hint, it's about 2x faster:

$ time convert -define jpeg:size=2048x1536 image.jpg -resize 1024x768 1024x768_image.jpg
real    0m0.195s
user    0m0.604s
sys 0m0.016s

You'll see a dramatic speedup for very large jpg files.

You could also consider another thumbnailer. vipsthumbnail, for example, is quite a bit faster again:

$ time vipsthumbnail image.jpg -s 1024x768 -o 1024x768_image.jpg
real    0m0.111s
user    0m0.132s
sys 0m0.024s

Although real time is down by only a factor of 2, user time is down by a factor of 5 or so. This makes it useful to run with gnu parallel. For example:

parallel vipsthumbnail image.jpg -s {} -o {}_image.jpg ::: \
    1024x768 250x250 174x174

The results does look appealing but unfortunately I was not able to compile the vips package on my Slackware system due to some library incompatibilities. Due to lack of time I did not pursue this solution but since numbers speak for themselves I will definitely return to this package sometime soon. Thanks for that! — Didzis, Feb 06 '17 at 10:28
There are some slackbuilds for vips, eg. https://github.com/igorek7/SlackBuilds/tree/master/vips though that one is a bit old. — jcupitt, Feb 06 '17 at 11:02

Mark Setchell · Answer 2 · 2017-02-03T17:14:55.563

Eric's and John's suggestions share much wisdom, and can be mixed in with my suggestion - which is to use GNU Parallel. It will REALLY count when you have thousands of images.

I created 100 images (actually using GNU Parallel, but that's not the point) called image-0.jpg through image-99.jpg. I then did a simple resize operation just to show how to do it without getting too hung up on the ImageMagick aspect. First I did it sequentially and it took 48 seconds to resize 100 images, then I did the exact same thing with GNU Parallel and came in under 10 seconds - so there are massive time savings to be made.

#!/bin/bash
# Create a function used by both sequential and parallel versions - it's only fair
doit(){
   echo Converting $1 to $2
   convert -define jpeg:size=2048x1536 "$1" -resize 1024x768 "$2"
}
export -f doit

# First do them all sequentially - 48 seconds on iMac
time for img in image*.jpg; do
   doit $img "seq-$img"
done

# Now do them in parallel - 10 seconds on iMac
time parallel doit {} "par-{}" ::: image*.jpg

Just for kicks - watch the CPU meter (at top right corner of the movie) and the rate the files pop out of GNU Parallel in the last 1/6th of the movie.

Bonzo · Answer 3 · 2017-02-03T12:47:52.513

It is funny as I did some conversions a while ago and found mpr was slower as well. Anyway try this:

$cmd = " convert "image.jpg" -colorspace rgb -coalesce \( -clone 0 -resize 1024x768> -write 1024x768_image.jpg +delete \)".
" \( -clone 0 -resize 250x250> -gravity center -extent 250x250 -write 250_wide.jpg +delete \) ".
" -resize 174x174> -gravity center -extent 174x174  null: ";
exec("convert $cmd 174x174_image.jpg ");

I notice you do not have a background colour for you extent.

You can also add a -define to your loop method check it out in this list: https://www.imagemagick.org/script/command-line-options.php#define jpeg:size=geometry only reads the amount of data you need to create the image without reading the whole image. You could probably also add it into your first line. -quality is for the output and will have no effect where you put it.

 $cmd = " convert "image.jpg" jpeg:size=1024x768 -colorspace rgb -coalesce \( -clone 0 -resize 1024x768> -write 1024x768_image.jpg +delete \)"..

I can never remember if it comes before or after the file name though

added -define jpeg:size=1024x768 and used -clone functionality - still no performance gain — Didzis, Feb 06 '17 at 09:28
@Didzis: You need to add `-define jpeg:size` **before** the input image - exactly as user894763 shows you. — Mark Setchell, Feb 07 '17 at 08:41

score 2 · Answer 4 · answered Feb 03 '17 at 16:04

Try Magick Persistent Cache image file format (.mpc) over Magick Persistent Registry (.mpr).

convert "image.jpg" -quality 85 -colorspace rgb myoriginal.mpc
convert myoriginal.mpc \
    -resize "1024x768>" \
    -write "1024x768_image.jpg" \
    -resize "250x250>" \
    -gravity center \
    -extent "250x250" \
    -write "250x250_image.jpg"  \
    -resize "174x174>" \
    -gravity center \
    -extent "174x174" \
    "174x174_image.jpg"

Which results in the following times when tested with 1.8mb jpeg.

real    0m0.051s
user    0m0.133s
sys     0m0.013s

It's true that this will take two commands (although could be simplified to one with -write ... +delete), but very little I/O cost after .mpc is loaded back into the image stack.

Or

You can probably skip .mpc all together with ...

convert "image.jpg" -quality 85 -colorspace rgb \
    -resize "1024x768>" \
    -write "1024x768_image.jpg" \
    -resize "250x250>" \
    -gravity center \
    -extent "250x250" \
    -write "250x250_image.jpg"  \
    -resize "174x174>" \
    -gravity center \
    -extent "174x174" \
    "174x174_image.jpg"

With results...

real    0m0.061s
user    0m0.163s
sys     0m0.012s

Converting to mpc first and using the command you proveded second in a loop gave me about 7 seconds performance boost (62seconds previously - 55 now). Still three separate commands does the job much more quicly despite all the tuning mentioned here in answers/comments — Didzis, Feb 06 '17 at 09:22

ltlBeBoy · Answer 5 · 2018-06-14T16:19:00.660

ImageMagick has a special resize operator variation named -thumbnail <geometry> for converting very large images to small thumbnails. Internally it uses

-sample to shrink the image down to 5 times the final height which is much faster than -resize if the thumbnail is much smaller than the original image. Because this operator uses a reduced filter set, the -filter operator is ignored!
-strip to remove all profiles from the image which are usually not required for thumbnails. This also further reduces the size of the resulting image file.
-resize to finally create the requested size and ratio

When it comes to creating thumbnails from JPEG images, then the special JPEG shrink-on-load option -define jpeg:size=<size> can be used instead, as stated out by user894763. Be aware that this option has to be specified immediately after convert, e.g.:

convert -define jpeg:size=<size> input-image.jpg ...

Anyway the -thumbnail operator can be specified additionaly then because it removes all profiles from the thumbnail image and thus reduces the file size.

Detailed information can be found in the ImageMagick usage documentation.

score 0 · Answer 6 · answered Jun 14 '18 at 17:03

I get about half the time using one command line in ImageMagick 6.9.10.0 Q16 Mac OSX starting with a 3 MB input JPG image.

list="image.jpg"
time for img in $list; do
convert "image.jpg" \
-coalesce -resize "1024x768>" +repage "1024x768_image.jpg"

convert "1024x768_image.jpg" \
-coalesce \
-resize "250x250>" \
+repage \
-gravity center \
-extent "250x250" "250x250_image.jpg"

convert "1024x768_image.jpg" \
-coalesce \
-resize "174x174>" \
+repage \
-gravity center \
-extent "174x174" "174x174_image.jpg"
done

Time: 0m0.952s

time convert "image.jpg" \
-resize "1024x768>" \
+write "1024x768_image.jpg" \
-resize "250x250>" \
-gravity center \
-extent "250x250" \
+write "250x250_image.jpg"  \
-resize "174x174>" \
-gravity center \
-extent "174x174" \
"174x174_image.jpg"

Time: 0m0.478s

The coalesce in your multiple commands is not needed, since JPG does not support a virtual canvas. So removing that reduces the multiply command line time to 0m0.738s

Multiple commands should be longer since you have to write and read intermediate images. Since your intermediate images are JPG, you will lose more visual quality each time you write and read the intermediate images. So the quality of one long command line should be better, also.

ImageMagick single convert command performance

6 Answers6

Linked