0

I have simple scrap application where I want to find all hrefs on matches:

public class MainClass {

    public static void main(String args[]) {
        WebDriver driver = null;
        driver = new HtmlUnitDriver(BrowserVersion.CHROME);
//        driver = new ChromeDriver();
        driver.manage().timeouts().pageLoadTimeout(5, TimeUnit.MINUTES);

        driver.get("https://www.tipsport.cz/live");
        System.out.println("HTMLELEMENT: \n" + driver.getPageSource());
        List<WebElement> elements = driver.findElements(By.xpath("//div[@id='events']//a"));
        System.out.println("Size of elements: " + elements.size());
    }
}

If I do this by this way via HTML unit driver I'm getting "Size of elements: 0". When I look into the consolse and try to find out element with id="events". It is there:

<div class="liveList" id="events">

When I run the same code with ChromeDriver, it finds elements, currently it was 51.

Why this row

List<WebElement> elements = driver.findElements(By.xpath("//div[@id='events']//a"));

doesn't do the same in ChromeDriver and HTMLUnitDriver, could you please help me what I'm doing wrong? I need to run it without chrome

Ahmed Ashour
  • 5,179
  • 10
  • 35
  • 56
user1604064
  • 809
  • 1
  • 9
  • 29
  • the site "https://www.tipsport.cz/live" seems to use some kind of js bootstrap. the difference between the HtmlUnitDriver and the Chromedriver is that the HtmlUnitDriver doesnt execute javascript thus the site looks different to the htmlUnitDriver as it does to the Chromedriver (tip: open the site with any normal browser with javascript off. Thats what you get when you use HTMLUnitDriver) – Tom K Jun 12 '17 at 11:36
  • So it seems that here could be the problem :/ I though that BrowserVersion.CHROME ensure the same behavior for javascript. So there is no way how to make the same via HTML unit driver? – user1604064 Jun 12 '17 at 12:03
  • Just a hint: since some months, `HtmlUnitDriver` enabled JavaScript by default. – Ahmed Ashour Jun 12 '17 at 12:13

1 Answers1

0

With direct HtmlUnit access (2.27-snapshot), and waiting, it retrieves the values:

try (final WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
    String url = "https://www.tipsport.cz/live";

    HtmlPage page = webClient.getPage(url);
    Thread.sleep(5_000);
    List<?> elements = page.getByXPath("//div[@id='events']//a");
    System.out.println(elements.size());
}

I am not sure which HtmlUnitDriver version you use, and why latest released version returns 0.

Ahmed Ashour
  • 5,179
  • 10
  • 35
  • 56
  • Hi Ahmed, I'm using org.seleniumhq.selenium selenium-htmlunit-driver 2.52.0. Does the HtmlUnit from net.sourceforge.htmlunit have the same features like HtmlUnitDriver from selenium? Like writing text to input box, clicking on the lement and submitting the forms? – user1604064 Jun 12 '17 at 12:47
  • You use an old `artifactId`, please check [here](https://github.com/SeleniumHQ/htmlunit-driver). Actually HtmlUnitDriver is on top of HtmlUnit, so you can directly use it. However if you plan to switch between drivers, then use HtmlUnitDriver/ChromeDriver/etc. – Ahmed Ashour Jun 12 '17 at 12:50