tricks for speeding up tar while tarring up a huge directory of little files?

Question

I'm trying to tar up a directory that has about 3M tiny files in it. Tar is chugging along, but I'm thinking its going to take longer than I can wait.

I'm wondering if telling tar to not store metadata (owner, group, perms) would reduce the churn on reading and re-reading this huge directory and maybe speed things up, and if there is a tar switch that does this.

My initial perusal of the man page only gets me something like --no-xattrs, which looks like a start, but I was hoping someone had some specific knowledge.

score 3 · Answer 1 · answered Jan 06 '11 at 16:35

3

I've got to think that the performance of the underlying filesystem probably has a lot to do with what you're getting out of tar.

What is the filesystem you're reading from, and how tiny are the files?

answered Jan 06 '11 at 16:35

Matt Simmons

20,396
10
68
116

score 1 · Answer 2 · answered Oct 14 '11 at 02:30

1

You might try adding mbuffer between your tar and your output file. I just learned about it, and it did wonders for my transfer.

e.g. tar -c dir | mbuffer > output.tar

Might not help if the slowness really is a a result of disk or filesystem limitations.

answered Oct 14 '11 at 02:30

Brad Langhorst

45
3

How do you envision this being faster than `tar -cf output.tar dir` ? – sciurus Oct 14 '11 at 02:52
mbuffer will buffer the output stream but tar does no calculating operations so what gain can be had ? – SvennD Nov 10 '17 at 09:57

tricks for speeding up tar while tarring up a huge directory of little files?

2 Answers2