2

How to sync many small files between two far away sites with limited slow network, and site B have old copy of site A. Can I use git to accelerate this work?

Usually I like use winrar to add site A data to pakage and robocopy to site B, but this way waste old copy on site B.

If use robocopy directly, the speed will awfull slow since so much small files.......

Now I know git can transfer files diffs with compress. Should I create a empty git repostitory on site B, and send to site A. Then add two site data to their git repo separately ? But I think the two git repository can't only transfer diff files when push back to site B. There must have a first whole copy transfer of site A git repo, isn't it? It's not a daily backup work, just one time.

How to only transfer diff files with most speed and compress..... Other software and suggestion is welcome, like caculate two site data's MD5 and compare to find out diff files, then pakage and compress to transfer..... Thank you....

Thanks. Since my data in on win2003 both sites, Should I install cygwin for tar rsync and gzip command? I try cygwin1.7, but the gzip is 1.4 and have not --rsyncable option..... Just tar | rsync will compare and compress? I read some windows cwrsync article, seems I have to install on rsync server and client,.....

Bill the Lizard
  • 398,270
  • 210
  • 566
  • 880
solaris2008
  • 21
  • 2

2 Answers2

2

rsync is more suitable for synchronizing files by transferring compressed deltas.

wRAR
  • 25,009
  • 4
  • 84
  • 97
  • I'm not sure rsync is more efficent than robocopy when dealling many small files.....? The data platform is win2003, and robocopy is also copy different files(maybe just check file size and date, I think robocopy not calculate MD5 because it have not clint/server.) – solaris2008 Sep 23 '11 at 08:21
1

Tar, gzip (with --rscynable) and rsync are your friends:

tar cf - /siteA | gzip -c --rsyncable > A.tar.gz
rsync -a A.tar.gz siteB:/somewhere
hroptatyr
  • 4,702
  • 1
  • 35
  • 38
  • Thanks. Since my data in on win2003 both sites, Should I install cygwin for tar rsync and gzip command? I try cygwin1.7, but the gzip is 1.4 and have not --rsyncable option..... Just tar | rsync will compare and compress? I read some windows cwrsync article, seems I have to install on rsync server and client,..... – solaris2008 Sep 24 '11 at 04:09
  • Yes, rsync has to be on both sides. Yes, just tar and rsync will work too, use rsync with the -z switch in that case (which will compress the stream). – hroptatyr Sep 26 '11 at 07:23