.NET WebRequest Throttling

Question

I'm using C# .NET 4 and MSSQL.

The code uses WebRequest to download the html of different websites. The code is multi threaded.

I want to limit the number of requests per mintue for a pre-defined list of domains.

For example

Domain: www.domain1.com Request Per Min : 5

Domain: www.domain2.com Request Per Min : 20

i saw some questions/topics that recommended to implement a "leaky bucket". I'm just not sure how to do that.

Thx!

score 0 · Answer 1 · answered Jul 01 '11 at 17:27

You could do something like this:

while (true)
        {
            // read how long to wait between scans
            // we do this here so that we could, conceivably, update this value while the scanner is running
            int scanwaittime = int.Parse(ConfigurationManager.AppSettings["WaitTime"].ToString());


            if (Engine.IsRunning)
            {
                // engine is already running another scan - give it some more time
                System.Threading.Thread.Sleep(scanwaittime);
            }
            else
            {
                ...
            }
        }

This should point you in the right direction, I think.

.NET WebRequest Throttling

1 Answers1