Replacement for FTP?

Question

I have some fairly large file transfers going across the atlantic every morning by ftp. From time to time the transfers will be distrupted, likely due connectivity issues, and the files will be left in an incomplete state.

I'm wondering if there is a better way to handle these file transfers. Would say, rsync or scp be more resilient to periodic loss of connectivity? The best would be if the transfer would reconnect and continue where it left off, should it be disrupted.

Large in this case means files up to several GiB and transfers spanning several hours.

Doesn't FTP do those things? Perhaps you need to tweak the settings on the client/server? — V. Romanov, Nov 03 '09 at 08:28
The common ftp clients doesn't do this on their own, but it might be possible to script a framework. Eg. read the exit code, if not 0 restart the transfer. — Roy, Nov 03 '09 at 08:40

score 10 · Accepted Answer · answered Nov 03 '09 at 08:29

10

I belevei this is a case where rsync should work well. Rsync does a very good job at resuming where it left off when a transfer is incomplete. If an older version of the file already exists on the remote side rsync is really good at only transmitting the differences.

I don't really think scp is what you want. If you need to transmit over ssh, then you can pass the option -e ssh to rsync and ssh will be used as the transport.

answered Nov 03 '09 at 08:29

Linux Geek

366
2
6

+1 for rsync. It happened to me yesterday when my wifi connection got cut off for a while. The transfer resumed after that. – sybreon Nov 03 '09 at 08:56

score 7 · Answer 2 · answered Nov 03 '09 at 09:47

7

I second the rsync option, however if you want it to resume a transfer of a big file, then don't forget to use the --partial command-line option which will mean it doesn't delete the temporary file it was writing to when the link goes down. You'll probably have to call it in a loop which retries when the link goes down.

Something like

until rsync --partial -e ssh sourcefile remote:destination; do 
 echo Retrying; 
done

answered Nov 03 '09 at 09:47

davidsheldon

401
3
7

Thats a very elegant loop! – Roy Nov 03 '09 at 11:09

score 2 · Answer 3 · answered Nov 03 '09 at 10:13

2

Why not BitTorrent? You get multiple bonuses if you use it. At least reliable transfers and availability to seed from multiple locations.

answered Nov 03 '09 at 10:13

Anonymous

1,550
1
14
18

1

It's a point to point transfer of private data. Bittorrent is good for many things, but probably not for this. – Roy Nov 03 '09 at 11:08
You can encrypt it before transferring. – Anonymous Nov 06 '09 at 16:52

score 1 · Answer 4 · answered Nov 03 '09 at 12:05

1

Definitely use rsync with -P - but for better performance over the Atlantic you probably want to use one of the UDP based file transfer tools - Aspera, FileCatalyst - or the open source Tsunami UDP.

answered Nov 03 '09 at 12:05

James

7,643
2
24
33

Thats an interesting idea. Even though I'd prefer to use standard utilities, I'll have a good look at these. – Roy Nov 03 '09 at 14:01

score 0 · Answer 5 · answered Nov 03 '09 at 12:33

0

Maybe Direct Connect (or other P2P protocol) is the best choice for you?

answered Nov 03 '09 at 12:33

SaveTheRbtz

5,691
4
32
45

I don't think a p2p file sharing solution is appropriate for these transfers. Thanks anyway. – Roy Nov 03 '09 at 14:02

score 0 · Answer 6 · answered Nov 09 '09 at 18:04

Have you checked out accellion? They are a ftp/webmail hybred. We use it for our graphics designers to securely and easily send large files to the printers.

They offer a free virtual appliance. It also has encryption, delivery notification and forwarding prevention, 2GB max file size, all built-in for free.

If you pay you can setup automation and allow up to 10GB files.

Replacement for FTP?

6 Answers6