0

I have a server for which I want to start doing some daily backups of certain files. Since server is in production, it is better to limit network bandwidth, I thought trickle would do the job, but it does not seem to have any effect at all and I don't understand why.

Server A has the files I want to archive. I created an NFS share on it and gave access to Server B where I mounted the files. On Server B I run this command:

trickle -s -u 20000 -d 20000 find /mnt/ServerA/log/ -newermt 20181001 -not -newermt 20181002 -type f -print0 | xargs -0 tar -cvzf /tmp/archive.tar.gz

When I observe Server B network traffic using iptraf-ng tool, I see both servers fully utilizing their 1 Gigabit bandwidth (they are located in same lan). trickle does not limit it to 20Mbps as I specified in the command above.

Here is what I see in iptraf-ng: 846.68 Mbps (Both Server A and B) while the command is running.

Why isn't trickle command working and what could I use instead that is relatively simple.

alexfvolk
  • 164
  • 2
  • 10
  • On a particular host, you can limit outbound, but not inbound. The inbound bandwidth has already been used by the time your host sees the traffic to limit it. The best you can do is police the traffic on Server B so that TCP (if it is TCP, nothing to be done about UDP) will react to all the lost traffic by slowing down, but remember that TCP will constantly try to speed up again, so it is somewhat inexact. What you really want to do is limit outbound on Server A. Server A is going to try to send as fast as possible unless you tell it otherwise, but not on Server B. – Ron Maupin Nov 13 '19 at 00:21
  • @RonMaupin this is only partially true - especially on the above case. You can limit inbound traffic by signalling TCP to slow down by dropping packets. In a single stream scenario this is fairly effective – davidgo Nov 13 '19 at 00:52

1 Answers1

1

I am not very familiar with trickle, but it appears to me that in your command, trickle is operating on the "find" command rather then the tar command which would be where the bandwidth is in use.

I would look at rejigging the command to use PV to limit the speed of the stream. I've not tried it, but I imagine something like this would work:

  find /mnt/ServerA/log/ -newermt 20181001 -not -newermt 20181002 -type f -print0 | tar -c --files-from - | pv --rate-limit 2m | gzip > /tmp/archive.tar.gz

Breaking this down, the find command finds your files as before. the --files-from allows tar to read the files into a stream (uncompressed). The pv command acts as a break to streaming - with 2m being 2 megabytes, so a little more then 20 megabits. This tar stream is then compressed into a gzip file.

I note here that a lot of extra fluffing arround went into streaming and then compressing the file - this is because I understand the source is the remote system. If the source were the local system and the file were being written to a remote one you could use a much simpler variant with tar doing the gzipping to a stream and then piping that through pv and writing that out - but that will limit the speed of the compressed stream.

davidgo
  • 6,222
  • 3
  • 23
  • 41
  • Now I have more questions :P I tried pv without any parameters and got a 10MiB/s transfer rate: # pv /var/log/testfile/testfile1GB.txt | nc -w 1 10.10.0.19 5201 ^C30MiB 0:01:01 [10.1MiB/s] [====================> ] 63% ETA 0:00:35 But when I just cp the same file from a NFS share, I get close to 900mbps. Why the variation if I did not give a limit to pv? – alexfvolk Nov 14 '19 at 20:26
  • Not sure. What happens if you play around with the -I parameter for NC? I do wonder if netcat is line buffered and if that might be contributing to the speed difference. – davidgo Nov 15 '19 at 20:19