How can I compress jpegs in Python more efficiently?

Question

I'm working with thousands of large image files in a regularly updated library. The following script does the job (on average reduces my file size ~95%) but costs me around 25 seconds to compress one image. Obviously, I can just let the script run overnight, but it would be cool if I can shave some time off this process. I'm mostly looking for any unnecessary redundancies or overhead in the script that can be trimmed out to speed up the process. I'm still new to Python, so go easy on me.

from PIL import Image
from pathlib import Path
import os, sys
import glob

root_dir = "/.../"

basewidth = 3500

for filename in glob.iglob(root_dir + '*.jpg', recursive=True):
    p = Path(filename)
    img = p.relative_to(root_dir)
    new_name = (root_dir + 'compressed/' + str(img))
    print(new_name)
    im = Image.open(filename)
    wpercent = (basewidth/float(im.size[0]))
    hsize = int((float(im.size[1])*float(wpercent)))
    im = im.resize((basewidth,hsize), Image.ANTIALIAS)
    im.save(new_name, 'JPEG', quality=40)

Thanks!

If you have working code and you're looking for improvements, a better place to ask might be https://codereview.stackexchange.com/ — Mike Scotty, Jan 21 '20 at 21:09
You can process multiple files in parallel using [the `multiprocessing` module](https://docs.python.org/3.8/library/multiprocessing.html#module-multiprocessing) — ForceBru, Jan 21 '20 at 21:10
PIL isn't known for its speed. You might look for a different library. — Mark Ransom, Jan 21 '20 at 22:05
You can do them very fast, in parallel, in a one-liner from the Terminal if that's an option? Tag me by my name with an `@` in front if so, and I will get notified that you have answered my comment. — Mark Setchell, Feb 04 '20 at 10:46
@MarkSetchell I'd be interested in hearing more about your terminal command — cburchie, Feb 05 '20 at 18:34

score 1 · Answer 1 · edited Feb 06 '20 at 07:17

As I mentioned in the comments, you can do this without Python, just in the Terminal with ImageMagick, which is included in most Linux distros and is available for macOS and Windows.

It looks like you want to convert a bunch of JPEGs to a width of 3,500 pixels at a quality of 40, correct?

Make a testing directory with 100 or so JPEGs in it - a COPY of non-precious files that you can play around and experiment on. This command should convert one single JPEG, so try that first:

magick input.jpg -quality 40 -resize 3500x result.jpg

If that looks correct, this command will do exactly the same to all the JPEGs in the current directory and save the results in a subdirectory called processed:

mkdir processed
magick mogrify -path processed -resize 3500x -quality 40 *.jpg

If that looks correct, we can parallelise it next.. let me know how it goes.

You can also do it with libvips. So, to process one image:

vipsthumbnail input.jpg -o result.jpg[Q=40] --size 3500x

And try that on all 100 images with:

for f in *.jpg ; do
    echo "Processing $f"
    vipsthumbnail "$f" -o processed/"$f"[Q=40] --size 3500x
done

If that is faster, we can pursue that in parallel.

If you are on macOS, you can install ImageMagick and vips with homebrew:

brew install imagemagick
brew install vips

Hey Mark, I swapped "vips" for "vipsthumbnail" in your example, hope that's OK. vipsthumbnail accepts the 3500x syntax, so it's a better match for this problem. — jcupitt, Feb 06 '20 at 07:18

How can I compress jpegs in Python more efficiently?

1 Answers1