Parsing dependent requests in an http web response

Question

I want to simulate the behaviour of the WebTestRequest class (in Visual Studio's Test Tools framework) where it can invoke dependent requests based on resources that are referred to in the response that is obtained from the original request.

For example, if I issue a web request and get the response by doing this:

string url = "http://www.mysite.com";
WebRequest request = WebRequest.Create(url);
using (WebResponse response = request.GetResponse())
{
    StreamReader reader = new StreamReader(response.GetResponseStream()); 
    string responseText = reader.ReadToEnd();
}

I would like to be able to parse responseText and see if there are any requests to other resources (like js/css files, images, etc.)

Is there an easy way of doing this? I hesitate to manually do this, as some of the resource requests may be set up programmatically and may not be obvious on a straightforward text parse.

score 0 · Answer 1 · answered Dec 30 '09 at 17:36

0

Use a html/sgml parser library. I'm not familiar with Visual Studio, but there are frameworks for parsing html out there. Find one and look in the API for something related to finding elements.

answered Dec 30 '09 at 17:36

dlamotte

6,145
4
31
40

Just to be clear, I'm already using the IHTMLDocument3 interface to build up a document object for html element extraction. I'm more interested in figuring out what dependent requests I should execute, based on the response. Do you know of an html parser that would enable me to do this? – Tola Odejayi Dec 30 '09 at 18:03
Sounds like you want a full-fledged web engine. Webkit is such a thing. If you REALLY need that, which I'm not sure you do, then you could look into that. Sorry I can't be of any more help. – dlamotte Dec 30 '09 at 18:37

agentnega · Answer 2 · 2012-05-09T06:01:18.223

I'm reasonably certain that WebTestRequest itself only does a "straightforward text parse" to determine the dependent requests, since it has no awareness of javascript. So if you were to implement such then your code would be accurately simulating the behaviour.

The following is a list of all the elements I could find in a cursory glance of the HMTL 4 spec that can refer to additional resources and thus would need to be parsed:

<link href=
<img src=
<script src=
<iframe src=
<object data=
<area href=

Not sure if it's exhaustive or not.

By the way, I'm curious as to what you ended up doing in the end.

EDIT:

some of the resource requests may be set up programmatically and may not be obvious on a straightforward text parse

It in fact becomes impossible at some point to determine the dependent requests from parsing an html response, and I'll give you an example: anything developed with Google Web Toolkit. In a recent GWT app that I tested, there was essentially no parsable html -- everything is run from javascript. Extracting obvious path names (when available) wasn't even useful because in reality conditional logic was selecting certain dependents and not others.

Parsing dependent requests in an http web response

2 Answers2