Can you "Queue" webRequest to the background to speed up your program?

Question

The slowest part of my code are these lines right here:

        string responseString;
        var webRequest = WebRequest.Create(url);
        using (var response = webRequest.GetResponse())
        using (var sr = new StreamReader(response.GetResponseStream()))
        {
            responseString = sr.ReadToEnd();
        }
        return responseString;

is there a way to "queue up" multiple webRequest and have them download in the background while my code parses the HTML between page requests? Is there a name/method for this I can google? any links or code are much appreciated! thanks!

*edit: full getHTML() code added

You can go [async](http://msdn.microsoft.com/en-us/library/86wf6409(v=vs.110).aspx) — rene, Jul 22 '13 at 09:18

score 3 · Answer 1 · answered Jul 22 '13 at 09:24

You can use the async methods, to unblock your UI and handle the requests there.

But your posted line of code should do almost nothing. The request to the server occurs when calling the Response. With:

var respone = webRequest.GetResponse();

Just use the, to unblock the UI and handle it after the async progresss ist done.

var response = webRequest.GetResponseAsync();
// or BeginResponse() ... EndResponse();

You can read the following for further information: http://msdn.microsoft.com/en-us/library/86wf6409(v=vs.110).aspx

Feel free to extend my answer to your hearts delight. – Rand Random Jul 22 '13 at 20:21 — Rand Random, Jul 22 '13 at 20:21

score 1 · Answer 2 · answered Jul 22 '13 at 20:22

What you're looking for is a producer consumer style of programming. Whenever you see that the BlockingCollection is often a good tool. It's a collection (by default using a queue) that is designed to be accessed from multiple threads safely.

You can start out with your consumers that are asynchronously making the requests and then putting the results of the request into the queue:

BlockingCollection<string> queue = new BlockingCollection<string>();

foreach(string url in links)
    var webRequest = WebRequest.Create(url);
    webRequest.GetRequestStreamAsync()
        .ContinueWith(t => new StreamReader(t.Result).ReadToEnd());

Then you can have code that processes each of the results in the queue as they come in: (There can easily be more than one consumer if that makes sense as well.)

foreach(var fileText in queue.GetConsumingEnumerable())
{
    processText(fileText);
}

Can you "Queue" webRequest to the background to speed up your program?

2 Answers2