2

For a long time i've been trying to emulate web browser using HttpWebRequest. Using most known or basic headers like : User Agent - Accept(content type) - Protocol Version - Accept (language) - Referer - Method - Keep Alive.. and for sure CookieContainer.

yet still with many tests page content may differ when using a Real Browser.

So, my question is: How to completely Emulate a web browser using HttpWebRequest or at least be close to it?

what other headers am i missing? Is there any other stuff i should use beside headers and cookie container ?

Thanks.

Dr.Vision
  • 85
  • 1
  • 5
  • 14
  • You can only do so much that way. Most pages have client-side javascript and other content that won't get processed. – Crowcoder Jan 15 '15 at 17:56
  • and what about pages that doesn't have any javascript content? – Dr.Vision Jan 15 '15 at 17:59
  • Trace with Fiddler how a browser makes the request to the page in question, and then use the same request (headers, cookies, etc.) with HttpWebRequest. Or better yet HttpClient if you're using the latest version of .Net. If the content is different then the browser is doing something. Browsers are a lot more complicated than you might think and they don't just display the response verbatim. – Crowcoder Jan 15 '15 at 18:04
  • i did use fiddler and still no difference in headers!! – Dr.Vision Jan 15 '15 at 18:15
  • Like I said, browsers don't just display the response verbatim, they have complex rendering logic. The result of HttpWebRequest goes through none of that. You might be able to pass your response through a rendering engine but I wouldn't have experience with that. – Crowcoder Jan 15 '15 at 18:21
  • 1
    @Dr.Vision maybe, the keyword you should google is *headless browser* – EZI Jan 15 '15 at 18:49

0 Answers0