how much multiprocess can speed up on compress task?

Question

I have a large compress task (compress 10000 large file( > 10 G each))

I found xz -z or gzip are slow.

So, i want to use multiprocess to make the compress parallelized.

but i heard the high IO task's limit is on IO speed.

I am not so good in hardware design.

So, can multiprocess speed up compress?

Have a look here... https://stackoverflow.com/a/34166037/2836621 — Mark Setchell, Feb 28 '21 at 08:36
Hey to use `fxz` or `zpaq` both work with several threads. For compression or decompression. — TTho Einthausend, Feb 28 '21 at 11:35
If you're doing 100TB of disk reading, and several TB of writing, I hope your disk subsystems are well configured. — Mark Setchell, Feb 28 '21 at 12:16

score 1 · Accepted Answer · answered Feb 28 '21 at 16:01

1

Yes, it can. Compression is usually compute bound, not I/O bound. You can use pigz to get close to a factor of n speedup for n cores.

answered Feb 28 '21 at 16:01

Mark Adler

1 Answers1