404 - not found. But browser can read the url

Question

In my ASP.Net app, I open a request to a url as follows:

webRequest = WebRequest.Create(url);
response = webRequest.GetResponse();

But the GetResponse returns an exception of:

The remote server returned an error: (404) Not Found.

The url does in fact exist. If I copy the url and enter it in my browser, it will show the page. What could be causing this exception?

What is the value of `url` in your code? – Hariprasad Dec 20 '13 at 06:10 — Hariprasad, Dec 20 '13 at 06:10
Try WebRequest to webclient – Ramesh Rajendran Dec 20 '13 at 06:25 — Ramesh Rajendran, Dec 20 '13 at 06:25

score 4 · Accepted Answer · answered Dec 20 '13 at 06:11

4

Depends on the URL.

Some sites are configured so that if they don't get a recognized user-agent, they'll assume the site is being crawled by an unauthorized bot and will either return a 403 Forbidden or a 404 Not Found, depending on the site.

answered Dec 20 '13 at 06:11

RomSteady

398
2
13

Investigating further, I discovered that while it does return a 404, it also returns a response with the data. ASP.Net seems to ignore the content and generate an exception. Is there something else other than WebRequest that can be used where no exception gets generated when the response is read and where I can read the actual content's body? – Johann Dec 20 '13 at 06:18
Catch the System.Net.WebException, and read from the exception's Response.GetResponseStream() method? – RomSteady Dec 20 '13 at 06:31

404 - not found. But browser can read the url

1 Answers1