tar and ssh large file

Question

I have running the following command for 5 hours and still going. mysql has gigabytes of data, but how can I make sure that the following command is actually transferring data. Can someone check the correctness?

tar -czpf - ./ --exclude mysql mysql-bin* mysql.sock | ssh -lroot xxx.xx.xxx.xx tar -xzpf - -C /var/lib/mysql

May I ask, why you don't use `rsync`? – polemon Jan 30 '11 at 08:14 — polemon, Jan 30 '11 at 08:14

score 13 · Accepted Answer · answered Jan 30 '11 at 00:59

You can insert the pv command into your pipeline to get a report of how many bytes have been transferred, like this:

tar -czpf - ./ --exclude mysql mysql-bin* mysql.sock | 
pv |
ssh -lroot xxx.xx.xxx.xx tar -xzpf - -C /var/lib/mysql

This will give you output like this, including the total number of bytes and the current transfer rate:

 202MB 0:00:13 [  17MB/s] [            <=>                                    ]

You can also just run du -sh on the destination directory.

score 0 · Answer 2 · answered Jan 30 '11 at 13:18

If you have a fast network between the two server, you should try without compression (-C).

To limit the use of CPU, you should change the default encryption algorithm used with "-c blowfish" for instance.

(you can also use ifstat or iptraf to check speed of data exchange)

tar and ssh large file

2 Answers2