Bandwidth throttling while copying files between computers

Question

I've been trying to make a program to transfer a file with bandwidth throttling (after zipping it) to another computer on the same network.

I need to get its bandwidth throttled in order to avoid saturation (Kind of the way Robocopy does).

Recently, I found the ThrottledStream class, but It doesn't seem to be working, since I can send a 9MB with a limitation of 1 byte throttling and it still arrives almost instantly, so I need to know if there's some misapplication of the class.

Here's the code:

using (FileStream originStream = inFile.OpenRead())    
using (MemoryStream compressedFile = new MemoryStream())      
using (GZipStream zippingStream = new GZipStream(compressedFile, CompressionMode.Compress))
{
    originStream.CopyTo(zippingStream); 
    using (FileStream finalDestination = File.Create(destination.FullName + "\\" + inFile.Name + ".gz"))
    {
        ThrottledStream destinationStream = new ThrottledStream(finalDestination, bpsLimit);
        byte[] buffer = new byte[bufferSize];
        int readCount = compressedFile.Read(buffer,0,bufferSize);
        while(readCount > 0)
        {
            destinationStream.Write(buffer, 0, bufferSize);
            readCount = compressedFile.Read(buffer, 0, bufferSize);
        }
    }
}

Any help would be appreciated.

Where did you find this `ThrottledStream`? It's not in the standard library.. — Blorgbeard, Jun 06 '13 at 21:59
It's not a standar library. Found it right here. http://www.codeproject.com/Articles/18243/Bandwidth-throttling — Francisco Palma, Jun 06 '13 at 22:20
I've checked the `ThrottledStream` code... looks like it might work as advertised, but I can see a couple potential problems with it. Most importantly though, how many bytes are actually being written to it, and how many at a time? To me it appears that `ThrottledStream` simply delays each write operation *after the first*. If all of your data is written in the first write, it goes through immediately. *Subsequent* writes are delayed, but each write is sent in one burst after a delay. — Corey, Jun 07 '13 at 00:21

score 1 · Answer 1 · answered Jun 07 '13 at 00:34

The ThrottledStream class you linked to uses a delay calculation to determine how long to wait before perform the current write. This delay is based on the amount of data sent before the current write, and how much time has elapsed. Once the delay period has passed it writes the entire buffer in a single chunk.

The problem with this is that it doesn't do any checks on the size of the buffer being written in a particular write operation. If you ask it to limit throughput to 1 byte per second, then call the Write method with a 20MB buffer, it will write the entire 20MB immediately. If you then try to write another block of data that is 2 bytes long, it will wait for a very long time (20*2^20 seconds) before writing those two bytes.

In order to get the ThrottledStream class to work more smoothly, you have to call Write with very small blocks of data. Each block will still be written immediately, but the delays between the write operations will be smaller and the throughput will be much more even.

In your code you use a variable named bufferSize to determine the number of bytes to process per read/write in the internal loop. Try setting bufferSize to 256, which will result in many more reads and writes, but will give the ThrottledStream a chance to actually introduce some delays.

If you set bufferSize to be the same as bpsLimit you should see a single write operation complete every second. The smaller you set bufferSize the more write operations you'll get per second, the smoother the bandwidth throttling will work.

Normally we like to process as much of a buffer as possible in each operation to decrease the overheads, but in this case you're explicitly trying to add overheads to slow things down :)

Thanks. It worked! :). I also changed the copy code with a CopyTo and now it transfers with the appropiate throttling. — Francisco Palma, Jun 07 '13 at 16:10
I also found out that the throttling process doesn't always work as desired, because from time to time (not regularly) the stream skips the throttling. — Francisco Palma, Jun 10 '13 at 16:30
@FranciscoPalma Glad you got it working. I'm mildly surprised that CopyTo works... will have to look into how it is writing the data to see if it's using small blocks, since that could lead to a fairly large overhead on some operations. Good to know anyway. — Corey, Jun 10 '13 at 23:13

Bandwidth throttling while copying files between computers

1 Answers1