WebRequest bug?

Question

EDIT: Solved, the problem was server-side.

I'm using C# and .NET2 and I wonder is that a WebRequest bug.. I do several good requests with this method and all is fine, but after that every time I get "The operation has timed out.". I really don't understand why is that.

public string RequestPage(string url) {
        HttpWebRequest req = null;
        string line = "";
        string site = "";

        try {
            req = (HttpWebRequest) WebRequest.Create(url.Trim());
            req.Timeout = 10000;

            StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream());
            while ((line = reader.ReadLine()) != null) {
                site += line;
            }

            return site;
        } catch (Exception ex) {
            MessageBox.Show("ERROR " + ex.Message);
        }

        return null;
    }

No, this is not a bug in `HttpWebRequest`. What happens if you try to paste the url in your browser? — Darin Dimitrov, Aug 20 '10 at 13:31
Yes, but probably you need to send some HTTP headers for this to work like `User-Agent` and `Accept`. Have you verified what HTTP header does your browser send? There are sites that block bots that do not identify themselves. — Darin Dimitrov, Aug 20 '10 at 13:33
Its possible that YouTube is throttling your requests because it believes that you are a bot. Are you making requests very frequently? — Justin, Aug 20 '10 at 13:34
It's also possible that the HttpWebRequest is clogging up resource because you don't dispose it. — dtb, Aug 20 '10 at 13:36
Darin Dimitrov: They are valid for some time. All is fine for 5-6 requests (everytime different number) Kragen: maybe. but what is 'valid' for youtube? I'm requesting video pages, nothing special. — blez, Aug 20 '10 at 13:36
@David Christiansen, `HttpWebRequest` doesn't implement `IDisposable` so wrapping it in a `using` won't compile. — Darin Dimitrov, Aug 20 '10 at 13:44
Indeed. I just tried using() with my stream, bug I have the same bug. — blez, Aug 20 '10 at 13:45

score 2 · Answer 1 · answered Aug 20 '10 at 13:43

2

You're not disposing of the response:

using (WebResponse response = req.GetResponse())
using (StreamReader reader = new StreamReader(response.GetResponseStream())
{
    while ((line = reader.ReadLine()) != null) {
        site += line;
    }
}

Basically there are pooled connections per server that you talk to. You're running out of them because you're not closing the response. The above should sort it out.

Additionally:

That's a potentially very slow way of building up a string. Use a StringBuilder to concatenate text content in a loop.
Do you really want to remove all the line breaks? If not, just use reader.ReadToEnd() instead.

answered Aug 20 '10 at 13:43

Jon Skeet

1,421,763
867
9,128
9,194

@blez, the code that @Jon showed is perfectly fine so try it with some other site. If it works then youtube is throttling your requests. – Darin Dimitrov Aug 20 '10 at 13:52
@Darin Dimitrov yes, it works for other sites. So it should be youtube. What should I do? Send IE/Firefox headers or? – blez Aug 20 '10 at 13:57
@blez, yes, try sending FireFox headers. If this doesn't work you are out of luck. Maybe look for some API that youtube exposes. I am not aware, never worked with it. – Darin Dimitrov Aug 20 '10 at 13:58
I just used Firefox header, and the problem persists. BUT.. All works fine after a restart of the app. So the problem should be from my side? – blez Aug 20 '10 at 14:05
@blez, could you post the url you are testing with? – Darin Dimitrov Aug 20 '10 at 14:08
I use these links: http://pastie.org/1104156 and every time after the 3-4th link it fails until restarting. – blez Aug 20 '10 at 14:10

dtb · Accepted Answer · 2010-08-20T13:52:55.663

I don't know if this solves your problem, but you should always dispose a HttpWebResponse (and other objects that implement IDisposable) when you're done:

public string RequestPage(string url)
{
    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
    req.Timeout = 10000;

    using (WebResponse resp = req.GetResponse())
    using (StreamReader reader = new StreamReader(resp.GetResponseStream()))
    {
        return reader.ReadToEnd();
    }
}

If you don't actually require all the features of HttpWebRequest, you can use WebClient instead:

public string RequestPage(string url)
{
    using (WebClient client = new WebClient())
    {
        return client.DownloadString(url);
    }
}

Using the your second method it sill hangs. Maybe it's some youtube bot protection? — blez, Aug 20 '10 at 13:54

WebRequest bug?

2 Answers2