2

rsync is much slower than expected in my use case:

I'm facing the problem of frequently copying multiple hundred huge media files (each way bigger than 100GB) from a Synology NAS to a local Thunderbolt RAID via LAN using a Mac. I've tried many different options ranging from Finder to rsync. None of which seem to work great (or even good).

Surprisingly just copying using Mac OS Finder is the fastest option that I tested so far. It averages at about 100MB/s. But for many reasons (error handling, no checksums, .DS-Store problems, etc) this method is not at all satisfactory.

Although I do not need to sync anything (just copy entire directories from A to B), rsync has exactly what I need in terms of features but it seems to be way too slow for this kind of task. It averages at about 50-70MB/s (-vaEWh --progress).

My question is: Am I holding it wrong or is rsync not the right tool for this? What else could be used to copy many very large files at speeds above 100MB/s with a decent amount of monitoring and logging?

Thanks for any helpful tips!

nimuc
  • 21
  • 2
  • 1
    Probably just as a curio, this is the fastest way to copy over tcp as far as I know: on dest start to listen and untar data `nc -l 9999 | tar xvf -`, on the source side `tar cvf - . > /dev/tcp/10.10.0.2/9999` (change dest ip) if source host does not have bash something like `tar cvf - . | nc 10.177.59.7 9999` – kwarunek Jan 12 '16 at 00:42
  • I don't have an `rsync` man page handy. Could you edit your Q to show what `-vaEWh` are meant to do? Some of them may be "expensive:"options. Also, you haven't mentioned anything about your network specs. Is this at home, or in a large commercial environment, Speed? Other competing tasks? I'd be surprised if you find anything significantly faster the rsync, it is a long established and well supported tool. (As I can't duplicate this issue, I won't have much else to offer on this topic). Good luck! – shellter Jan 12 '16 at 10:14
  • I've tested rsync with just -Wr without any improvements. So the selected options are not slowing it down. Network: Mac to GBE Switch to NAS. No other active tasks. The Finder is twice as fast as rsync. The Finder is able to get to 100MB/s while rsync only gets to 50-70MB/s. – nimuc Jan 12 '16 at 12:57
  • @shellter explainshell is a *great* site for quickly seeing what options are what on a copy-and-paste shell command: https://www.explainshell.com/explain?cmd=rsync+-vaEWh – Jim K. Sep 14 '17 at 19:16
  • @JimK. Thanks, I didn't realize that explainShell went into that much detail! Great to know. Thanks!!! – shellter Sep 14 '17 at 20:29

1 Answers1

2

Apparently the rsync that Apple ships has some known issues with speed. While I haven't tried it others are reporting that installing rsync through ports is resolving the issue.

http://www.laszlopusztai.net/2015/02/19/fixing-extremely-slow-rsync-on-os-x/

Brian Monroe
  • 21
  • 2