Data throttling C#

Question

I am using thirdparty component which accepts maximum 25KB data at a time. I am passing array of the objects to this third party component from my application.

However the amount of data that my application writes is much more than 25KB. I am retrieving data from database and calling the component directly.

I have added reference of the component in the application. The data I am passing to the component as array of object which contains primitive and non primitive types.

How can I implement the data throttling here?

Please add some more info, like how is the data being sent to the component, i.e. what sort of communication is it. — Boris B., Aug 27 '12 at 09:24
Also, how are you determining if an 'object' is more than 25KB ? — Dominic Zukiewicz, Aug 27 '12 at 09:29
@DominicZukiewicz: I assume that this is exactly the question OP wants to have answered. I would simply measure(f.e. with `GC.GetTotalMemory`) how many objects you can store in the array until the `25KB` exceeds, then i would query and send only that array. — Tim Schmelter, Aug 27 '12 at 09:30
@TimSchmelter: 'GC.GetTotalMemory()' will not be a good representation because if you create a stream, that will create internal buffers, which may take you over the 25KB limit. There may also be some internal allocations you may not be aware of. I'll add an answer to this effect. — Dominic Zukiewicz, Aug 27 '12 at 09:36
@DominicZukiewicz: That was just my first thought and my intention was not to use it during streaming but once to measure the number of objects he can store before it exceeds the limit. (obsolete anyway since he mentioned that it's no an array of primitives) — Tim Schmelter, Aug 27 '12 at 09:40

score 5 · Accepted Answer · answered Aug 27 '12 at 09:25

You can calculate the size of the one row in your database. After that every time you pass something you add that size to a variable. At the same time you are using a Stopwatch that runs. Just check if the Stopwatch.EllapsedSeconds is bigger than 1 second. If yes reset the Stopwatch and reset your variable with the size you ve already passed. If not check if the size you ve already passed (the amount of variable) is bigger than 25KB. If that is true call System.Thread.Thread.Sleep(Math.Max(1000 - StopWatch.EllapsedMilliseconds, 0)). But remember you have to do that in an extra thread because sleep blocks the whole thread.

Boris B. · Answer 2 · 2012-08-27T09:38:21.337

You must implement a data buffer between your app and the component. The best way to do that is to:

make a class that has it's own internal thread,
on the public part of the class' interface implement Write method which accepts a byte array and stores it to a queue
internal thread writes up to 25kb chunk of data from queue the and sleeps for 1 second, minus the time it took to write the chunk.

You also must take care, if the data stream is continuous and is produced at a rate > 25kbps, then you produce more then the component can consume and your queue will overflow.

score 2 · Answer 3 · answered Aug 27 '12 at 09:40

Have a look at the BufferedStream class example in MSDN. You should be able to adapt it so that you write 25KB maximum per second.

When writing to stream, you specify a data size to read/write, so if you say write 25KB and it returns 24KB, you know you can still write 1K.

If it comes back with 25KB and there is still data outstanding in the buffer, you can do a Thread.Sleep(1000) to ensure it does not exceed the quota.

Data throttling C#

3 Answers3