2

We use rsync as backup on a NFS share we have on our VPN network.

The ISP's modem is awful and it stops responding when we ask too much upload bandwidth to it. We have 150Mbps down / 20 Mbps up according to speedtest.

The number of files and size varies greatly, rsync checks 200.000 files, in one of the backups, of very variate sizes, from KB to 50 GB or more.

To avoid this, we use rsync's bwlimit option. The problem is that apparently even 10s bursts can make the modem irresponsive.

From its manual entry:

Due to the nature of rsync transfers, blocks of data are sent, then if rsync determines the transfer was too fast, it will wait before sending the next data block. The result is an average transfer rate equaling the specified limit.

Example of crontab line for the backup of /var/www

flock rsync_wan_lock -c "nice -n1 ionice -c2 -n2 rsync -aq --partial --bwlimit 1000K --delete-after /var/www  /mnt/foo_bk/."

Is there any way to mitigate or solve this?

Andrew Schulman
  • 8,811
  • 21
  • 32
  • 47
quimnuss
  • 155
  • 8
  • Can you prodive the `rsync` options you use? What is the available upload bandwidth? What bwlimit value did you try? How many files are you uploading? What is the total size of the files? – Gohu Jan 11 '18 at 11:54
  • updated the question with the info you asked. – quimnuss Jan 11 '18 at 12:24
  • You say "in one of the backups", are there multiple backups that could occur in parallel? (several lines in cron may occur simultaneously). Also, does lowering the bwlimit value fix the issue? – Gohu Jan 11 '18 at 13:11
  • There are multiple backups but they can't run in parallel as per the `flock` (that's the reason behind it). Lowering the `bwlimit` makes the problem come less frequently, but it's hard to tell, I'm unsure how the size of the blocks written is decided by rsync. – quimnuss Jan 15 '18 at 09:37
  • Have you tried configuring the limit on the VPN tunnel level? – Gerald Schneider Jan 15 '18 at 09:43
  • No, and it's a good idea. I'll check if it's possible. – quimnuss Jan 15 '18 at 10:19
  • VPN specific, there isn't. There's BWM, but setting it doesn't limit bandwidth. It might be that it only comes into action when prioritizing traffic since the default rule is `no limit`. We have a Zywall 110. – quimnuss Jan 15 '18 at 14:06
  • Well, the only way I found to workaround this is to switch to rsync's ssh copy instead of NFS sharing. That means I had to open ssh on the backup server and set the keys. It's not a big deal since that server is only visible from within the VPN network. In short, let rsync handle the bandwidth transfer instead of the NFS filesystem. – quimnuss Jan 16 '18 at 11:20
  • You can try with the [trickle](https://unix.stackexchange.com/a/224413/50687) tool. – A.L Nov 24 '21 at 00:34

0 Answers0