0

I have to backup 500GB using the traditional tar -cvf command (without any compressions) However tar is adding each file one by one and this way it's taking a very long time

Is there any way to make tar multi threaded (like it can add multiple files to the tar archive at once)

Mario
  • 148
  • 7
  • You can't make `tar` itself multithreaded, but please add some more details to understand if/how we can make it faster. What specific command are you using? Are you tarring from/to the same disk? To another disk? To a NFS mount? – shodanshok Nov 10 '21 at 08:18

2 Answers2

2

No, and you're unlikely to be CPU bound in this case anyway.

Improve your backup strategy if you can, or live with slow backups but the only way to speed things up with the mechanism you have now is faster disks.

Ginnungagap
  • 2,595
  • 10
  • 13
  • *the only way to speed things up with the mechanism you have now is faster disks* Maybe not. If the backup process is, ummm, not well-thought-out and is doing things like wasting disk IO cycles with a process like "tar the file in place, then copy it to the backup media", it can easily be made 3-4 times faster by simply doing "tar the file directly to the backup media". The data being backed up should come off disk and go directly to the backup media - anything else is, ummm, poorly-designed. – Andrew Henle Nov 08 '21 at 00:28
  • Sure if the tar operation itself is slow, it will certainly help to send the data to another disk (or network or pigeon or whatever) so the disk can focus on reading, but it'll still be a largely IO bound process. I take your point about poorly designed backups though – Ginnungagap Nov 08 '21 at 00:35
0

CPU matters, you can try parallel tar with pigz.

As mentionned by /u/Andrew Henle

compressing on the source then transport could be more efficient than compressing on the fly to the destination.

For example :

time tar -I pigz -cf sourcebackup_$(date +"%Y-%m-%d-%H-%M").gz sourcebackup/

can be faster than :

time tar -I pigz -cf - /sourcebackup | ssh -t user@backupserver "sudo cat > /backup/sourcebackup_$(date +"%Y-%m-%d-%H-%M").tar.gz"
proxyd43
  • 152
  • 1
  • 2
  • 12