7

I have several PNG images in a directory and I'm using optipng to optimize and reduce image size. The problem is that it takes too long to optimize all files.

I have a quad core processor and I noticed that optipng was using only a single core when I optimize the directory.

This is the code I'm using:

ls -1 | while read line
do 
    optipng -o7 "$line"
done

Is it possible to execute optipng for four different files in parallel while reading a directory?

Denilson Sá Maia
  • 47,466
  • 33
  • 109
  • 111
Niresh
  • 611
  • 5
  • 16

3 Answers3

11

There is another solution involving xargs.

find some/dir/ -iname '*.png' -print0 | xargs -0 -n 1 -P 4 optipng -o7

Where -P 4 launches 4 parallel processes and -n 1 uses at most one filename per process.

Alternatively, if you have newline-separated filenames, use:

find some/dir/ -iname '*.png' | sort | xargs -d \\n -n 1 -P 4 optipng -o7

Thanks to Joe Lencioni comment on a blog.

Update: I've written a shell script to call zopflipng (which gives a higher compression than optipng) for several images in parallel: zopflipng_in_place

Denilson Sá Maia
  • 47,466
  • 33
  • 109
  • 111
  • Just out of curiosity, why would you want to limit it to one filename per process? – UpTheCreek Jun 26 '15 at 07:55
  • 1
    @UpTheCreek: `optipng` supports multiple filenames, so I could use that. However, assigning one file per process should result in a better distribution. Consider 2 parallel processes (each on a single CPU core), and consider 4 files: 2 very slow (very large) and 2 very fast. If I assign 2 files per process, it may happen that the 2 very slow files get queued on the same CPU core, while the other core will end up idle. Assigning 1 per process should not leave any CPU core idle until no files are remaining. – Denilson Sá Maia Jun 26 '15 at 09:21
  • 1
    Just tried a few experiments, and got much better performance passing high n values (50 in my case). With `-n 50 -P 3` I was getting solid 100% cpu on 3 cores. With `-n 1 -P 3` the CPU usage was all over the place (presumably due to the higher overheads of setting up/tearing down more processes). – UpTheCreek Jun 26 '15 at 10:11
  • One nice thing I noticed about not-too-high n values though, is that you can monitor the progress through large filesets in `htop`, so maybe something like `-n 10` would be the sweet spot for me. – UpTheCreek Jun 26 '15 at 10:21
  • @UpTheCreek: What kind of files? Small PNG images or large ones? For large ones (near 1MB or more), I believe `-n 1` will be better. For very small ones, I agree with you, larger `-n` will be better. – Denilson Sá Maia Jun 26 '15 at 15:09
  • The pngs were about 100KB before optimisation. – UpTheCreek Jun 30 '15 at 07:18
5

I use optipng with GNU parallel (included in every linux distribution):

parallel --bar 'optipng {}' ::: file1.png file2.png morefile*.png

The advantage: You have a bar indicating the progress.

erik
  • 2,278
  • 1
  • 23
  • 30
  • 1
    @MarcusJ The names `file1.png file2.png morefile*.png` where just example filenames. Replace them by your files names. If you have to many files at the time (I think it was more than 4000), you have to do it in several steps. Move part of the files to different subdirectories. – erik Aug 28 '17 at 20:25
  • install with `sudo apt install parallel` – rubo77 Mar 17 '21 at 13:55
4

You need to put each optipng in the background to use the four cores, and use a counter n to keep track of their number. I'm using n=4 in this case, so there'll be 4 background jobs run at a given time:

n=0
for image in *
 do
  optipng -o7 "$image" &
  n=$(( $n + 1 ))
  [ "$n" -eq 4 ] && n=0 && wait
done

Don't run many bg jobs simultaneously (keep n low) or there'll be a performance penalty. Modify the code as needed, especially the * pattern in for image in *.

lind
  • 2,159
  • 1
  • 12
  • 5