Why does a bad url result in UnknownError, instead of a 400 or 404 code? WebException

Question

Why does this not create WebExceptionStatus.ProtocolError? Instead I get WebExceptionStatus.UnknownError.

string url = "http://www.ogggle.com/"; // a bad url

var request = (HttpWebRequest) WebRequest.Create(url);

...options setup ellided

using (var response = (HttpWebResponse) request.GetResponse())
    {
        foreach (Cookie cookie in response.Cookies)
            cookies.Add(cookie);

        string urlContent = "";
        using (var sr = new StreamReader(response.GetResponseStream()))
            urlContent = sr.ReadToEnd();

        return urlContent;
    }

which will now throw a WebException; however WebExceptionStatus is actually UnknownError. I am expecting a ProtocolError.

catch (WebException we)
   {
    string message = "WEB EXCEPTION DETECTED. Retry counter: " + _retryCount;
    message += Environment.NewLine + "Problem with url: " + url + ". Status is: " + we.Status;
    message += Environment.NewLine + "Message is: " + we.Message;

    if (we.Status == WebExceptionStatus.ProtocolError)
    {
        message += Environment.NewLine + string.Format("Status Code : {0}", ((HttpWebResponse)we.Response).StatusCode);
        message += Environment.NewLine + string.Format("Status Description : {0}", ((HttpWebResponse)we.Response).StatusDescription);
    }

    WriteToErrorLog(message);
    if (DEBUG)
        Console.WriteLine(message);
    _retryCount++;
   }

From my perspective, the WebException should be smarter than this. A malformed or bad url should result in a meaningful WebExceptionStatus. Instead it results in the less-than-helpful UnknownError. Can anyone help me to understand how to get a meaningful WebException when the the url is a bad one?

HTTP status codes come from the HTTP response. If the URL cannot be parsed into a hostname, there is no HTTP response. — SLaks, Dec 02 '14 at 21:42
Have you tried to read the response `var resp = new StreamReader(we.Response.GetResponseStream()).ReadToEnd();` ? Maybe it can help. — L.B, Dec 02 '14 at 21:42
@L.B i am already calling urlContent = sr.ReadToEnd(); which is what I think you are suggesting. — sapbucket, Dec 02 '14 at 22:07
@sapbucket No, I suggested it to use it in your expection block (See the variable `we`, its your WebException). — L.B, Dec 02 '14 at 22:18

score 0 · Answer 1 · answered Dec 02 '14 at 21:55

0

You do not provide enough information in your question to figure out what the problem is. What is the real "bad" URL that you are having a problem with? WebRequest.Create(string) calls new Url(string), which will throw a UriFormatException exception when the given string is not a valid URL. That's what it does with your given code and the string .

answered Dec 02 '14 at 21:55

Jonathan Amend

12,715
3
22
29

I'm using a url string generator that permutes over 1000's of possible url string combinations. About 10% of them are valid. I then blast a target website to grab data from the valid pages. I don't know if they are valid until after i send them through the response process. For some reason I get the behavior that I wrote about above. – sapbucket Dec 02 '14 at 22:03
After changing the originally given url from "aBadUrl" to "http://www.ogggle.com/", it now gives a `WebException` with status `NameResolutionFailure`, which seems reasonable and is still not what is described in the question. – Jonathan Amend Dec 02 '14 at 22:16
That is exactly what I want to happen. Thank you for confirming. I'm going to debug it once again and see if I can reproduce what you are seeing. – sapbucket Dec 03 '14 at 21:35

Why does a bad url result in UnknownError, instead of a 400 or 404 code? WebException

1 Answers1