why HTML Agility Pack HtmlDocument.DocumentNode is null?

Question

I'm using this code to change the href attribute of a HTML stream.

first I download a full html page using this code:(URL is webpage address)

HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(URL);
HttpWebResponse myHttpWebResponse = 
                         (HttpWebResponse)myHttpWebRequest.GetResponse();

Stream s = myHttpWebResponse.GetResponseStream();

then I process this:

HtmlDocument doc = new HtmlDocument();

doc.Load(s);
foreach (HtmlNode link in doc.DocumentNode.SelectNodes("/a"))
{
    string att = link.Attributes["href"].Value;
    link.Attributes["href"].Value = "http://ahmadalli.somee.com/default.aspx?url=" + att;
}
doc.Save(s);

s is html stream.

but I've got an exception that says doc.DocumentNode is null!

i tried many sites but doc.DocumentNode is null to

Is it pointing to a full HTML document, or an HTML fragment? — Oded, Feb 04 '12 at 07:36
And what is the structure of the HTML? Does is have an `HTML` element? A `BODY` element? Is it a valid document? — Oded, Feb 04 '12 at 08:56
@ahmadali shafiee, I cannot reproduce your error. `doc.DocumentNode.SelectNodes` could return `null` if no nodes found. It seems your xpath expression doesn't found any node and return null. Could you paste a full code which throws a `NullReferenceException` exactly on `doc.DocumentNode`? — Oleks, Mar 03 '12 at 16:22
@Alex: It is a method that loads a webpage and I tried many webpage addresses whit the code. in all cases the `doc.DocumentNode` is null! — ahmadali shafiee, Mar 03 '12 at 16:26
@ahmadali shafiee: sorry, but I can't see how `doc.DocumentNode` could be `null` with the scenario you provided. — Oleks, Mar 03 '12 at 16:54

score 8 · Accepted Answer · edited Mar 07 '12 at 09:53

8

This works for me.

using(WebClient client = new WebClient())
{
    client.Encoding = System.Text.Encoding.UTF8;
    var doc = new HtmlAgilityPack.HtmlDocument();
    doc.LoadHtml(client.DownloadString("http://www.google.com?q=stackoverflow"));
    foreach (var href in doc.DocumentNode.Descendants("a").Select(x => x.Attributes["href"]))
    {
        if (href == null) continue;
        href.Value = "http://ahmadalli.somee.com/default.aspx?url=" + HttpUtility.UrlEncode(href.Value);
    }
    StringWriter writer = new StringWriter();
    doc.Save(writer);
    var finalHtml = writer.ToString();
}

Also see the HttpUtility.UrlEncode to be able to get the url back correctly. Otherwise, some parameters in original url may cause problem.

Use HttpUtility.UrlDecode to decode it.

edited Mar 07 '12 at 09:53

ahmadali shafiee

4,350
12
56
91

answered Mar 03 '12 at 17:45

L.B

114,136
19
178
224

I tried your code but I've got an exception:`Object reference not set to an instance of an object.` – ahmadali shafiee Mar 05 '12 at 10:21
@ahmadalishafiee I run it on another machine. It worked without a problem. – L.B Mar 05 '12 at 12:29
First: I tried it using [this link](http://google.com) it worked fine! the I tried it using [this](http://codeplex.wordpress.org) and got a NullRefrenceException. `href` in foreach statement is null! – ahmadali shafiee Mar 06 '12 at 11:17
I added `if (href == null) continue;` into the loop. – L.B Mar 06 '12 at 11:41

Cristian Lupascu · Answer 2 · 2012-02-04T09:58:31.977

0

Try using //a instead of /a.

In XPath, this basically means give me all the links in the document, as opposed to give me all the links in the document root.

Update:

The following code works fine:

        var myHttpWebRequest = (HttpWebRequest)WebRequest.Create("http://google.com");
        var myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();

        var s = myHttpWebResponse.GetResponseStream();

        var doc = new HtmlDocument();

        doc.Load(s);
        foreach (var link in doc.DocumentNode.SelectNodes("//a"))
        {
            var att = link.Attributes["href"].Value;
            link.Attributes["href"].Value = "http://ahmadalli.somee.com/default.aspx?url=" + att;

            Console.WriteLine(link.Attributes["href"].Value);
        }

edited Feb 04 '12 at 09:58

answered Feb 04 '12 at 09:05

Cristian Lupascu

39,078
16
100
137

anyway doc.DocumentNode is null and the exception thrown before of the foreach – ahmadali shafiee Feb 04 '12 at 09:11
Why do you feel the need to use `DocumentNode` at all? Why not use `doc.SelectNodes` directly? – Oded Feb 04 '12 at 09:15
@Oded: SelectNodes is a method of DocumentNode – ahmadali shafiee Feb 04 '12 at 09:36
@ahmadalishafiee Can you give an example of URL for which this fails? Or just a basic structure of the HTML? – Cristian Lupascu Feb 04 '12 at 09:42
@w0lf: I tried your suggestion but after that I just got a empty Stream and my new Stream was null – ahmadali shafiee Feb 04 '12 at 09:49
@ahmadalishafiee Try using `http://google.com` as the URL instead of just `google.com` (if you didn't do that already) – Cristian Lupascu Feb 04 '12 at 09:56
@ahmadalishafiee I've added a code sample that works (enumerates all links on the page) – Cristian Lupascu Feb 04 '12 at 09:59
@w0lf: first: I'm using http but stackoverflow comment doesn't show it. second: I tried your code but still an empty html like last try – ahmadali shafiee Feb 04 '12 at 11:07
@ahmadalishafiee Try using a different URL or try to diagnose your network connection (see if you need to configure a proxy) – Cristian Lupascu Feb 04 '12 at 11:20
@w0lf: It's a asp.net and runs into my browser and I can connect to stackoverflow. the problem isn't from network connection – ahmadali shafiee Feb 04 '12 at 11:38
@w0lf: there is no proxy or internet problems in my connection. – ahmadali shafiee Feb 17 '12 at 14:45
1

@ahmadalishafiee It's probably worth noting that Google will / does put temporary blocks on IP's if there are an above normal number of calls. This, might be causing you to get nothing returned from goog if you've been running the code a lot in a short space of time. You should probably try using some other urls for now, so as to eliminate that as a factor. – Phill Healey Feb 25 '14 at 21:45

score 0 · Answer 3 · edited May 23 '17 at 12:06

0

Here is your answer: HTML Agility Pack Null Reference.

edited May 23 '17 at 12:06

Community

1
1

answered Mar 03 '12 at 15:15

PraveenVenu

8,217
4
30
39

score 0 · Answer 4 · answered Mar 05 '12 at 04:23

Try using the below code:

HtmlDocument htmlDoc = new HtmlDocument
        {
            OptionAddDebuggingAttributes = false,
            OptionAutoCloseOnEnd = true,
            OptionFixNestedTags = true,
            OptionReadEncoding = true
        };
        try
        {
            using (Stream reader = myHttpWebResponse.GetResponseStream())
            {
                reader.Seek(0, SeekOrigin.Begin);
                htmlDoc.Load(reader, true);
            }
            HtmlNode node = htmlDoc.DocumentNode;
            if (node != null)
            {
                foreach (var href in doc.DocumentNode.Descendants("a").Select(x =>x.Attributes["href"]))
                 {
                     href.Value = "http://ahmadalli.somee.com/default.aspx?url=" +HttpUtility.UrlEncode(href.Value);
                 }
            }
        }
        catch { }

I am using HtmlAgility pack version: 1.4.0

Solved your problem? If no, please comment. Else mark as answer.

score 0 · Answer 5 · answered Mar 06 '12 at 08:58

Anchor tag reference is an incorrectly escaped string:

...doc.DocumentNode.SelectNodes("/a")    //incorrect
...doc.DocumentNode.SelectNodes("//a")   //correct
...doc.DocumentNode.SelectNodes(@"/a")   //also correct

The original code fails to select any nodes and evaluates to null; this should be checked against to prevent failing on, say, a document where there are no links at all (however unlikely that is :)

var anchors = doc.DocumentNode.SelectNodes("//a");
if (anchors != null)
{
    foreach (HtmlNode link in anchors)
    {
        /*do stuff*/
    } 
}

There is no difference in resulting strings between `"/a"` and `@"/a"`. — user7116, Nov 07 '12 at 15:21

why HTML Agility Pack HtmlDocument.DocumentNode is null?

5 Answers5

Linked

Related