25

I unzip using this :

unzip -q "file.zip" -d path

How can I unzip faster with utilizing multiple cores and threads?

Thanks

Gediz GÜRSU
  • 555
  • 4
  • 12
ale8530
  • 317
  • 1
  • 4
  • 9
  • 1
    Could you please explain a bit, what do you mean by multi-core here? – RavinderSingh13 Feb 16 '18 at 08:58
  • Why is multicore an issue here ? Do you mean you have big zip files and want to use multi threading to speed up the process ? – Aserre Feb 16 '18 at 08:59
  • 1
    Interesting read: https://www.quora.com/How-can-I-leverage-multiple-cores-while-unzipping-a-zip-file-Linux – hek2mgl Feb 16 '18 at 09:00
  • I would like to be able to tell unzip to use 4 CPUs, for example is it possible? – ale8530 Feb 16 '18 at 09:01
  • 2
    Following the argumentation in the article I've linked, even if it would be possible, you won't gain so much extra speed. – hek2mgl Feb 16 '18 at 09:02
  • hek2mgl It does not solve my problem, I would like to use the same tool – ale8530 Feb 16 '18 at 09:16
  • you can use **pigz** https://zlib.net/pigz/ a multithread implementation of gzip both when compressing and decompressing. Since gzip works on a single file when compressing a directory (eventually with subdirectories) you have to first make a `tar` archive. – Paolo Nov 10 '22 at 08:04

2 Answers2

12

In short: No, unzipping with multiple cores is not available.

The decompression normally has lower CPU-intensity than the compression (where multiple cores are often involved).

You wouldn't have much of an advantage anyway as the read/write-operations are more of the bottlenecks during decompression.

Stefan M
  • 868
  • 6
  • 17
  • Ok thanks for your explain :) – ale8530 Feb 16 '18 at 10:30
  • 23
    NVMe SSDs are super fast, but unzipping a 6gb file still takes ages. Pretty sure the CPU is the bottleneck. What gives? – Alexandre G May 07 '19 at 03:21
  • 1
    Using https://github.com/lukehutch/quickunzip unzipping Xcode10.2.zip takes 2m26s, using Archive Utility on Mac - 3m55s – Alexandre G May 07 '19 at 05:10
  • 1
    It is the CPU bottlenecking it, 1 core will always be maxed out newer NVMe based drives, indicating the bottleneck – Alex Jun 12 '19 at 20:31
  • try ripunzip: https://medium.com/@adetaylor/does-parallel-unzipping-work-cdaf89124c88 (I built static version for local testing https://github.com/hemnstill/StandaloneTools/releases/tag/ripunzip-0.4.0) its x2 - x4 times faster then unzip – hemn Jul 25 '23 at 07:03
0

thy pigz which takes advantage of the multi cores and unzips in multiple threads

akiva
  • 2,677
  • 3
  • 31
  • 40
  • 6
    as per [the man page](https://linux.die.net/man/1/pigz), pigz doesn't use multi threading for decompression – Aserre Feb 16 '18 at 09:02
  • 7
    Still better than unzip. `pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances.` – Andrea Bergonzo Nov 28 '19 at 03:59