3

I am migrating to a new server and, as well as html/php files, have a directory which contains around 90,000 files totalling 24GB in size which needs to be moved.

When I did a test migration, I used tar to create a tarball then wget on the new host and then extracted the tar file but, whilst it worked fine, it took around 3 hours all in to complete. That would 3 hours of downtime whilst I did this on the actual migration to ensure no new files come in or files were changed etc.

I am now planning the actual migration and am trying to find quicker ways of doing this and wondered about using rsync - I have only ever used rsync locally and only for a small number of files so would running rsync against 90k items be quicker than the above method?

I am not so worried about CPU, memory or network usage as long as the actual process completes quicker as this will be done out of hours when the system is quieter anyway.

bhttoan
  • 650
  • 3
  • 15
  • 27
  • 2
    just a hint, you can check the -z option of rsync, which compress the data when transferring. you can also check gere : http://superuser.com/questions/291803/best-way-to-copy-millions-of-files-between-2-servers – Pierre-Alain TORET Nov 01 '15 at 16:28
  • Instead of tar and wget you can transfer while tarring. Something like `ssh "tar czf - /mystuff" | tar xzf -`. It would only be bad if the transfer got interrupted. And you should first find out where your bottleneck is so that you can decide on how and if to compress. – Carsten S Nov 01 '15 at 23:11

4 Answers4

5

You will have to test to find out. There are many variables such as the speed of your storage system.

Consider restoring your tar archive or backup while the old system is up. Then during the downtime use rsync to copy the remaining changes. It still has to check many file modify times which takes time, but there is much less I/O and network transfer.

This incremental copy is what rsync is good at. Doing it in one go may not be faster than other tools.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34
  • Unfortunately the previous archive is not available as the target was reimaged after testing was completed - if I would have to test to know then I will use the current method as I know it works and then, for the future, will run some tests and compare as may be useful for a future DR scenario – bhttoan Nov 01 '15 at 18:16
  • Presumably you care about this data and have a backup, so there is an excellent restore test opportunity. Also, slow, simple and tested can be better than fast, clever and risky. Depends on your requirements including size of downtime window. – John Mahowald Nov 02 '15 at 20:35
4

Rsync may allow you to sync the files without taking the server down. For example, you could do a sync while in production, then take the server down, sync again to make sure you got everything that changed in the mean time, and then turn on the new server. The second sync will be incremental, and would take a fraction of the time of the whole, minimizing the downtime.

0

rsync will copy only diffs - only changes from source to target server. If you already have copy of your data on new server, rsync will only copy what has been changed between your tar and actual state.

Ondra Sniper Flidr
  • 2,653
  • 12
  • 18
0

If you want to do this with tar and not rsync, you can paralelize your tasks :

source :

tar ... | nc host port

destination :

nc -l -p port | tar ...

It will at least reduce your time by two

mick
  • 735
  • 6
  • 7