Is gunzip file1.gz file2.gz faster than looping over the files and gunziping each

Question

I have a process that iterates through a list of files calling gunzip on each of them. Wondering would it be faster to just call the gunzip once on all the files. How does gunzip file1.gz file2.gz actually operate and is there a faster method for when you want to unzip multiple large files?

score 0 · Answer 1 · answered Aug 29 '18 at 17:49

0

That would be a useless micro-optimization.

The time for looping is dwarfed compared to the time of decompressing, especially if like in your case you unzip large files.

Do whatever is more readable. Note that looping has no problem when the number of files is huge, while a single gunzip command might run into command line limits and require xargs(1).

answered Aug 29 '18 at 17:49

Jens

69,818
15
125
179

i know the looping time is tiny but it is significant if one decompression doesn't start until the previous one finishes. however im not sure passing gunzip multiple files at once would make this happen in parallel or not – emc211 Aug 29 '18 at 18:21

score 0 · Answer 2 · answered Aug 29 '18 at 19:53

You can do them all in parallel with GNU Parallel like this:

parallel gunzip {} ::: *.gz

That will unzip them N at a time, where N is the number of CPU cores you have.

If you want a progress bar:

parallel --bar gunzip {} ::: *.gz

If you want to see what it would do, but without actually doing anything:

parallel --dry-run ...

Is gunzip file1.gz file2.gz faster than looping over the files and gunziping each

2 Answers2