0

I'm trying to use HtmlUnit to scrape the content of a Website. I am able to programmatically login to my account, but most of the content on the client-side is generated using Javascript (Angular JS). So my questions are:

  1. Can we use HTMLUnit to scrape this page using Angular JS as the scripting language.

  2. This is my code:

final WebClient webClient = new WebClient(BrowserVersion.CHROME);
WebClientOptions webClientOptions = webClient.getOptions();
webClientOptions.setJavaScriptEnabled(true);
final HtmlPage page1 = webClient.getPage(//url);
final HtmlForm form2 = page2.getFormByName("login");
final HtmlSubmitInput button = form2.getInputByValue("Sign In");
final HtmlTextInput textField = form2.getInputByName("email");
textField.setValueAttribute("//email");
final HtmlPasswordInput textField2 = form2.getInputByName("login_password");
textField2.setValueAttribute(//password); 
final HtmlPage page3 = button.click();
webClient.waitForBackgroundJavaScript(30000);

Thank you in advance for your help.

Ahmed Ashour
  • 5,179
  • 10
  • 35
  • 56
  • PS: I get a lot of javascript errors. –  Jul 15 '15 at 19:20
  • Please provide the URL you use, so others can see the errors – Ahmed Ashour Jul 15 '15 at 20:33
  • --Solved Hey thanks a lot for your reply Ahmed. I found a work around to my problem. –  Jul 15 '15 at 21:19
  • What would that workaround be, @Nachiket ? You should post an answer to your own question so others can benefit. – Michael Oryl Nov 02 '15 at 16:21
  • Hi @MichaelOryl: Well i was able to get to the login page and complete the login functionality by using the above code. –  Nov 03 '15 at 18:34
  • Hi @MichaelOryl: Well i was able to get to the login page and complete the login functionality by using the above code. But there was a problem. The website I was trying to use was using server side JS to load the content. So I dropped to idea of scrapping the page. When I said I found a work around to the problem, all I meant was that I was able to complete the login functionality by using the above code. –  Nov 03 '15 at 18:41
  • OK, thanks. Appreciate it. – Michael Oryl Nov 04 '15 at 15:04

0 Answers0