5

I am trying to get data from a page that dynamically appends data on load.

Specifications: HTMLUnit : version : 2.14

But I am not able to get the new page after scroll. I tried using various browser versions and all possible code changes. It will be great if anyone can let me know what I am doing wrong. Moreover document.documentElement.scrollTop; is always returning zero.

final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_24);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.waitForBackgroundJavaScript(60000);

HtmlPage page = webClient.getPage("http://www.snapdeal.com/products/mobiles-mobile-phones/?q=Brand:Samsung");
System.out.println(page.getTitleText());
final String pageAsXml = page.asXml();
System.out.println("Page1=\n" + pageAsXml);         
String s = "window.scrollBy(0, window.innerHeight);document.documentElement.scrollTop;";
ScriptResult sr = page.executeJavaScript(s);
JavaScriptJobManager manager = page.getEnclosingWindow().getJobManager();

while (manager.getJobCount() > 4) {
    System.out.println("SCript Job count = " + manager.getJobCount());
    Thread.sleep(1000);
}
System.out.println("Result= " + sr.getJavaScriptResult() + "\n");

HtmlPage page2 = (HtmlPage) sr.getNewPage();
if(page == page2)
    System.out.println("No difference");
else
    System.out.println("Page2\n" + page2.asXml());

Thanks & Regards

Reeni

Majid Laissi
  • 19,188
  • 19
  • 68
  • 105
ReeniMathew
  • 69
  • 5
  • So window.scrollBy(0, window.innerHeight);document.documentElement.scrollTop; java script code exist on page ?? – Kick Feb 24 '14 at 09:19
  • Hi Niks. It is browser specific.right? Shouldn't I be able to mimic browser events like this? Correct me if I am wrong. – ReeniMathew Feb 24 '14 at 11:27
  • I guess similar JS will be in all the browser.I am not 100% sure that what ever script u r calling will run.Can you do one thing divide the script into 2 parameter first window.scrollBy(0, window.innerHeight) & second document.documentElement.scrollTop – Kick Feb 24 '14 at 12:15
  • Tried that also Niks. It didnt work. I used both just to know what is the result of execution. – ReeniMathew Feb 24 '14 at 15:12
  • i tried and on my mac and i found different page,page after js call is black with title Twitter Follow button.Wat is ur expected output ? – Kick Feb 24 '14 at 18:23
  • It is giving a different page Nik. But for the example I provided, it should give a page with more products listed. But am not getting such a page. – ReeniMathew Feb 25 '14 at 03:39
  • so r u sure that JS script is proper ?? – Kick Feb 25 '14 at 06:46
  • Ya. If you try loading the same page in browser, you can see that onscroll is loading more data. – ReeniMathew Feb 25 '14 at 10:32
  • kk i actually dnt use this method till now,only have info that this method execute JS script.Can you try this method on any other page so to get better idea – Kick Feb 25 '14 at 10:41

0 Answers0