Checking if URL exists - HTTP Request always returns an exception

Question

There are numerous posts on how to check if a URL is valid. All of them feature basically the same code, which seems to work for everyone - not for me though and I don't get why.

    public static bool ifURLexists(string url) 
    {

         try 
         {
              var request = WebRequest.Create(url) as HttpWebRequest;
              request.Method = "HEAD";
              //response ALWAYS throws an exception
              using (var response = (HttpWebResponse)request.GetResponse()) 
              {
                return response.StatusCode == HttpStatusCode.OK;
              }
         }
         catch 
         {
            return false;
         }
    }

I have tested the method with parameters such as "http://www.nonexistingwebsiteblabla.com" and "http://www.google.com". No matter if I insert an existing or a non existing URL, I get a WebException at this line:

    using (var response = (HttpWebResponse)request.GetResponse())

Why could it be not working?

Don't use `try-catch` like that if you're interested in the exception. In this case you are, because you want to find out what it is. Ten to one it's a proxy authentication error of your corporation. — CodeCaster, Nov 17 '16 at 10:49
I've tried without try-catch too. It throws me a "System.Net.WebException". — , Nov 17 '16 at 10:54
Yes, and what is the exception message? There are many WebExceptions... — Equalsk, Nov 17 '16 at 10:54
That error suggests a network connectivity issue between this program and the internet. Given that your access in a browser clearly works (or how would you be here?) it seems that something like a firewall might be the most obvious culprit and is blocking just this application. Unfortunately we can't say, all I can guarantee you is that your code is perfectly fine and not at fault. Are you behind a proxy or corporate firewall? — Equalsk, Nov 17 '16 at 11:01
Try using a ping in a cmd.exe to see if you get a response. The website may be blocked in your webbrowser. Try manually using an IE and see if you get to site. Check in IE to see if site is blocked. Add to allowed sites. — jdweng, Nov 17 '16 at 11:03

Vishal Kottarathil · Accepted Answer · 2016-11-17T12:50:54.130

Check the status WebException.Status This will let you know what specific web exception has occured.

Update: Try change the request.Method = "HEAD"; to GET and try.

Try with a unavailable (404) url, compare the status. Check whether anything is blocking your request.

This is how i manage in my code, i am handling using only ftp specific status.'CommStatus' is an ENUM with error codes which is available in whole application.

catch (WebException ex)
        {
            FtpWebResponse response = (FtpWebResponse)ex.Response;               
          switch(response.StatusCode)
            {
                case FtpStatusCode.ActionNotTakenFileUnavailable:
                    return CommStatus.PathNotFound; 
                case FtpStatusCode.NotLoggedIn:
                    return CommStatus.AuthenticationError;
                default: return CommStatus.UnhandledException;

            }


        }

Below are the available Status of WebException.

CacheEntryNotFound
This API supports the product infrastructure and is not intended to be used directly from your code. The specified cache entry was not found.

ConnectFailure
This API supports the product infrastructure and is not intended to be used directly from your code. The remote service point could not be contacted at the transport level.

ConnectionClosed
This API supports the product infrastructure and is not intended to be used directly from your code. The connection was prematurely closed.

KeepAliveFailure
This API supports the product infrastructure and is not intended to be used directly from your code. The connection for a request that specifies the Keep-alive header was closed unexpectedly.

MessageLengthLimitExceeded
This API supports the product infrastructure and is not intended to be used directly from your code. A message was received that exceeded the specified limit when sending a request or receiving a response from the server.

NameResolutionFailure
This API supports the product infrastructure and is not intended to be used directly from your code. The name resolver service could not resolve the host name.

Pending This API supports the product infrastructure and is not intended to be used directly from your code. An internal asynchronous request is pending.

PipelineFailure This API supports the product infrastructure and is not intended to be used directly from your code. The request was a piplined request and the connection was closed before the response was received.

ProtocolError
This API supports the product infrastructure and is not intended to be used directly from your code. The response received from the server was complete but indicated a protocol-level error. For example, an HTTP protocol error such as 401 Access Denied would use this status.

ProxyNameResolutionFailure
This API supports the product infrastructure and is not intended to be used directly from your code. The name resolver service could not resolve the proxy host name.

ReceiveFailure
This API supports the product infrastructure and is not intended to be used directly from your code. A complete response was not received from the remote server.

RequestCanceled This API supports the product infrastructure and is not intended to be used directly from your code. The request was canceled, the WebRequest.Abort method was called, or an unclassifiable error occurred. This is the default value for Status.

RequestProhibitedByCachePolicy
This API supports the product infrastructure and is not intended to be used directly from your code. The request was not permitted by the cache policy. In general, this occurs when a request is not cacheable and the effective policy prohibits sending the request to the server. You might receive this status if a request method implies the presence of a request body, a request method requires direct interaction with the server, or a request contains a conditional header.

RequestProhibitedByProxy
This API supports the product infrastructure and is not intended to be used directly from your code. This request was not permitted by the proxy.

SecureChannelFailure
This API supports the product infrastructure and is not intended to be used directly from your code. An error occurred while establishing a connection using SSL.

SendFailure This API supports the product infrastructure and is not intended to be used directly from your code. A complete request could not be sent to the remote server.

ServerProtocolViolation This API supports the product infrastructure and is not intended to be used directly from your code. The server response was not a valid HTTP response.

Success This API supports the product infrastructure and is not intended to be used directly from your code. No error was encountered.

Timeout This API supports the product infrastructure and is not intended to be used directly from your code. No response was received during the time-out period for a request.

TrustFailure
This API supports the product infrastructure and is not intended to be used directly from your code. A server certificate could not be validated.

UnknownError
This API supports the product infrastructure and is not intended to be used directly from your code. An exception of unknown type has occurred.

More details here: https://msdn.microsoft.com/en-us/library/system.net.webexceptionstatus(v=vs.110).aspx

Thanks. The Status is "Name resolution failure" with both existing and non existing URLS. I find it hard to understand what that exactly means. — , Nov 17 '16 at 12:32

score 0 · Answer 2 · answered Oct 16 '19 at 11:21

0

Also you can use this option.

IPHostEntry ipHost = Dns.GetHostEntry(url);

answered Oct 16 '19 at 11:21

Caner

813
1
12
26

Checking if URL exists - HTTP Request always returns an exception

2 Answers2

Linked