HtmlAgilityPack & Selenium Webdriver returns random results

Question

I'm trying to scrape product names from a website. Oddly, I seem to only scrape random 12 items. I've tried both HtmlAgilityPack and with HTTPClient and I get the same random results. Here's my code for HtmlAgilityPack:

using HtmlAgilityPack;
using System.Net.Http;

var url = @"http://www.roots.com/ca/en/men/tops/shirts-and-polos/";
HtmlWeb web = new HtmlWeb();
var doc = web.Load(url, "GET", proxy, new NetworkCredential(PROXY_UID, PROXY_PWD, PROXY_DMN));
var nodes = doc.DocumentNode.Descendants("div")
            .Where(div => div.GetAttributeValue("class", string.Empty) == "product-name")
            .Select(div => div.InnerText.Trim())
            ;

[UPDATE 1] @CodingKuma suggested I try Selenium Webdriver. Here's my code using Selenium Webdriver:

IWebDriver chromeDriver = new ChromeDriver(@"C:\TEMP\Projects\Chrome\chromedriver_win32");
chromeDriver.Url = "http://www.roots.com/ca/en/men/tops/shirts-and-polos/";
var items = chromeDriver.FindElements(By.ClassName("product-name"));
items.Count().Dump();
chromeDriver.Quit();

I tried this code but still no luck. There are over 20 items on that page, but I seem to only get a random 12. How can I scrape all items on that site?

Answer 1

Since the v1.5.0-beta92,

HtmlAgilityPack has a FromBrowser method that allows you to wait until all elements you want are ready.

Documentation: http://html-agility-pack.net/from-browser

string url = "http://html-agility-pack/from-browser";

var web1 = new HtmlWeb();
var doc1 = web1.LoadFromBrowser(url, o =>
{
    var webBrowser = (WebBrowser) o;

    // WAIT until the dynamic text is set
    return !string.IsNullOrEmpty(webBrowser.Document.GetElementById("uiDynamicText").InnerText);
});
var t1 = doc1.DocumentNode.SelectSingleNode("//div[@id='uiDynamicText']").InnerText

var web2 = new HtmlWeb();
var doc2 = web2.LoadFromBrowser(url, html =>
{
    // WAIT until the dynamic text is set
    return !html.Contains("<div id=\"uiDynamicText\"></div>");
});
var t2 = doc2.DocumentNode.SelectSingleNode("//div[@id='uiDynamicText']").InnerText

Console.WriteLine("Text 1: " + t1);
Console.WriteLine("Text 2: " + t2);

The trick here is to find something that tells you when the page is ready since it's impossible for the library to know.

Answer 2

So there are a couple issues that prevent the count from being correct.

1. The page has a lazy loader. You have to scroll down to trigger the load of the items over 12.

2. The page uses AJAX calls to load the items over 12.

So, you need to navigate to the page, scroll to the bottom of the page, wait for AJAX to complete, and then scrape the page. The code below is tested and returns 20 items.

The script

String url = "http://www.roots.com/ca/en/men/tops/shirts-and-polos/";
driver.navigate().to(url);
JavascriptExecutor js = ((JavascriptExecutor) driver);
int height = 1;
int lastHeight = 0;
while (lastHeight != height)
{
    lastHeight = height;
    js.executeScript("window.scrollTo(0, document.body.scrollHeight);");
    height = (int) (long) js.executeScript("return document.body.scrollHeight;");
}

waitForJSandJQueryToLoad(10);

List<WebElement> products = driver.findElements(By.cssSelector("div.product-name"));
System.out.println(products.size());
for (WebElement e : products)
{
    System.out.println(e.getText());
}

Support function

public boolean waitForJSandJQueryToLoad(int timeOut)
{
    WebDriverWait wait = new WebDriverWait(driver, timeOut);

    ExpectedCondition<Boolean> jQueryIsLoaded = new ExpectedCondition<Boolean>()
    {
        @Override
        public Boolean apply(WebDriver driver)
        {
            return (Boolean) ((JavascriptExecutor) driver).executeScript("return (window.jQuery != null) && (jQuery.active === 0);");
        }
    };

    ExpectedCondition<Boolean> jsIsLoaded = new ExpectedCondition<Boolean>()
    {
        @Override
        public Boolean apply(WebDriver driver)
        {
            return (Boolean) ((JavascriptExecutor) driver).executeScript("return document.readyState == 'complete'");
        }
    };

    return wait.until(jQueryIsLoaded) && wait.until(jsIsLoaded);
}

Output

20
Rideau Flannel Shirt
Westridge Denim Shirt
Rideau Flannel Shirt
Riverside Plaid Shirt
Riverside Plaid Shirt
Heritage Peppered Polo
Heritage Peppered Polo
Heritage Peppered Polo
Cedar Jersey Polo
Cedar Jersey Polo
Hope River Shirt
Hawthorne Surplus Shacket
Acadian Linen Shirt
Camp Short Sleeve Shirt
Foxley Short Sleeve Shirt
Heritage Peppered Polo
Foxley Short Sleeve Shirt
Waterway Indigo Shirt
Waterway Indigo Shirt
Resolute Flannel Shirt

Answer 3

For most single page apps or pages that load content dynamically you better off using an actual browser to navigate the pages. I'd suggest looking into selenium for this type of setup.

https://www.nuget.org/packages/Selenium.WebDriver

Answer 4

As others said, the page from this site loads itself dynamically using some javascript, so the Html Agility Pack just gets the first items.

Web Scraping can be tough, especially with modern sites which use more and more javascript, and it's in general very specific to the target site (I'm not even talking about the legal issues..). You can use various techniques to determine how to get the information you require.

In this case, if you use any network analyzer, you'll quickly see the site uses an 'sz' (for Size I guess) query string parameter that allows you to specify the number of items you want.

So, just modify your url for this:

var url = @"http://www.roots.com/ca/en/men/tops/shirts-and-polos/?sz=9999";

and get any numbers of items you want.