3

I'm backing up and transferring files automatically each night via scp.

I realize I get packet-level checksums via the protocol itself, but I'm wondering what the best practice is to ensure that a transfer hasn't failed, or partially completed etc.

I could use rsync, but my needs are simple and I just want to ensure that:

  1. there wasn't an error in transmission
  2. retry if there was (so that I don't discover that I'm 4 days stale when I go to recover)
Flimzy
  • 2,454
  • 18
  • 26
Scott Klarenbach
  • 569
  • 2
  • 8
  • 20

2 Answers2

6

scp does not guarantee file integrity, so a checksum comparison between source and destination would be wise.

You mention your needs are simple, but rsync does not necessarily add complexity and offers significantly more features that will make you feel more comfortable that your synced data is legit.

rsync -ave ssh /your/source/file remotehost:/your/dest/file

If $? (exit status) is not 0 (success), then retry/alert/etc... If you are syncing entire directories, rsync is also much more suited for that, especially if you need to retry on failure, as incremental syncs will be significantly faster with each successive run.

loopforever
  • 1,185
  • 8
  • 11
  • 1
    Wait, if I understand scp correctly it appends a MAC or a signature to your messages (because SSH). So that provides integrity. So no need for a checksum – David 天宇 Wong Jul 01 '15 at 16:52
  • 1
    I can only suggest that this is a very narrow interpretation of the problem. SCP over an existing file and kill your SCP process half way through. Then ask the question: is the destination file the same as the source file? – loopforever Jul 02 '15 at 18:39
  • well if you kill your process half way through it's pretty obvious no? I can imagine SCP will return different messages if the file was correctly transfered or not. That should be enough. – David 天宇 Wong Jul 02 '15 at 18:50
  • No, not if it's a large file >1 TB and it dies at some random point. – Andor Kiss Feb 10 '21 at 14:06
  • The first line of this answer is outright wrong. `scp` guarantees file integrity by returning an error code. – Fadeway May 31 '23 at 10:27
2

I'm not quite sure how rsync doesn't align with your simple needs.

rsync -avPe ssh user@host:/that/dir /goes/here

if you really want to use scp, though, just loop your scp and test its exit condition.. something like..


#!/bin/bash
STATUS=1
while [ $STATUS -ne 0 ]; do
    echo "scp'ing..."
    scp -r thisstuff/ user@host:/overthere
    STATUS=$?
done

scp will return 0 on success, and > 0 on error.

MrTuttle
  • 1,176
  • 5
  • 5
  • Another thing that should be added to the loop was to (1) generate the sha1sum of the file before transfer and (2) within the loop if the status was 0 use ssh to do an sha1sum of the remote file and then exit the loop if the sha1sums were equivalent. – mdpc Jul 12 '11 at 22:52
  • I posted that more to demonstrate how needlessly complex loading those requirements onto scp would be when rsync basically solves the problem in one command... – MrTuttle Jul 12 '11 at 22:54
  • An apt demonstration that rsync *is* the simple option... – womble Jul 12 '11 at 23:25