0

I have script scheduled with cron that transfers a file around to other machines using scp filename user@host:~ and today have noticed that sometimes the md5 of the files don't match. Upon closer inspection it seems the file is corrupt.

My question is, what tool can I use to ensure a non-corrupt transfer?

James
  • 325
  • 2
  • 11
  • 22

2 Answers2

4

You can use rsync. Rsync is more reliable because:

rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking its whole-file checksum

This description is under the -c flag in man rsync. You don't have to actually use this option.

Luke
  • 1,932
  • 6
  • 22
  • 27
  • 1
    I dont' think this works as desired. "-c, --checksum:This changes the way rsync checks if the files have been changed and are in need of a transfer.Without this option, rsync uses a “quick check” that (by default) checks if each file’s size and time of last modification match between the sender and receiver."This seems to indicate that the -c flag tells rsync to use a cehcksum to determine if a file has changed. It does not appear to say that the file will be checksummed after transfer to ensure non-corruption. http://notemagnet.blogspot.com/2009/10/getting-started-with-rsync-for-paranoid.html – jdw Sep 28 '11 at 14:10
  • "Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking its whole-file checksum, but that automatic after-the-transfer veri-fication has nothing to do with this option's before-the-trans-fer "Does this file need to be updated?" check." - So either way, rsync is better for transfers. – Luke Sep 28 '11 at 14:20
  • Agreed. Rsync is the better tool for this, just the -c flag is not required for this solution. – jdw Sep 28 '11 at 14:25
0

You should use a strong hashing utility that use SHA256 or SHA512 algorithms. See: MD5 collision vulnerabilities

Mircea Vutcovici
  • 17,619
  • 4
  • 56
  • 83
  • 1
    Do you really consider MD5 collisions an issue when it comes to dealing non-intentional corruptions? – andol Sep 28 '11 at 15:32
  • You shouldn't state such things so imperatively I'd say. md5 is quite ok for variety of appliance. As they say the better is enemy of just good. Tools are to be chosen adequatively to goal. – poige Sep 28 '11 at 15:36
  • The question was how to *ensure* a non-corrupt transfer. Think that you are doing this for a plane computer. Would you wait for the first MD5 collision that will trigger a disaster? – Mircea Vutcovici Sep 30 '11 at 15:00