Exception while reading HTML page using c#

Question

I am using below code to read HTML code:

            string urlAddress = "http://google.com";
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
            HttpWebResponse response = (HttpWebResponse)request.GetResponse();
            if (response.StatusCode == HttpStatusCode.OK)
            {
                Stream receiveStream = response.GetResponseStream();
                StreamReader readStream = null;
                if (response.CharacterSet == null)
                    readStream = new StreamReader(receiveStream);
                else
                    readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
                string data = readStream.ReadToEnd();
                response.Close();
                readStream.Close();
            }

However I am getting exception:

The remote server returned an error: (407) Proxy Authentication Required

Also, I have made changes to my app.config file:

 <system.net>
    <defaultProxy useDefaultCredentials="true" />
  </system.net>

but still getting the same error.

However, when I open and page in IE, it opens without asking any credentials.

Did you try this : http://stackoverflow.com/questions/11148288/the-remote-server-returned-an-error-407-proxy-authentication-required — adityaswami89, Oct 17 '14 at 04:39
seems like it is not reading google.com, but when I tried different url, it works. weird though — Aquarius24, Oct 17 '14 at 04:59

score 0 · Answer 1 · answered Oct 17 '14 at 04:41

0

Use this code if you want download google page

  using (WebClient web = new WebClient())
     {
          string d=   web.DownloadString("http://google.com");
     }

answered Oct 17 '14 at 04:41

pradeep varma

136
3

score 0 · Answer 2 · answered Oct 17 '14 at 04:47

When I did my Google spider to get some searches, I actually had to implement a web browser control and use it to render results and appear to be 'human'. This way all requests could be timed and rendered to a page -- especially AJAX stuff which you can't render using HttpWebRequest. It is inconvenient, but 100% working.

Exception while reading HTML page using c#

2 Answers2