HTMLUnit not working with AngularJS

Question

According to https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot, with HtmlUnit (2.13) I am trying to create a snapshot for a webpage using AngularJS (1.2.1).

My Java code is:

WebClient webClient = new WebClient();

webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.setCssErrorHandler(new SilentCssErrorHandler());

webClient.getOptions().setCssEnabled(true);
webClient.getOptions().setRedirectEnabled(false);
webClient.getOptions().setAppletEnabled(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setPopupBlockerEnabled(true);
webClient.getOptions().setTimeout(10000);

webClient.getOptions().setThrowExceptionOnFailingStatusCode(true);
webClient.getOptions().setThrowExceptionOnScriptError(true);
webClient.getOptions().setPrintContentOnFailingStatusCode(true);

HtmlPage page = webClient.getPage(new WebRequest(new URL("..."), HttpMethod.GET));
webClient.waitForBackgroundJavaScript(5000);
String result = page.asXml();

Although webClient.getPage(...) does not throws any exception the result string still contains "unevaluated angular expressions" such as

<div>
    {{name}}
</div>

I am aware of http://htmlunit.10904.n7.nabble.com/htmlunit-to-scrape-angularjs-td29931.html#a30075 but the recomendation given there does not work either.

Of course the same GET-request works without exceptions in all current browsers.

Any ideas/experiences how to get HtmlUnit working with AngularJS?

Update:

I created a HTMLUnit bug report.
For the moment, I switched my implementation to PhantomJS. Maybe this code snippet helps others with a similar problem:

System.setProperty("phantomjs.binary.path", "phantomjs.exe");
DesiredCapabilities caps = new DesiredCapabilities();
caps.setJavascriptEnabled(true);
caps.setCapability("takesScreenshot", false);

PhantomJSDriver driver = new PhantomJSDriver(caps);
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
driver.get(new URL("..."));
String result = driver.getPageSource();

Update2: I stoped rendering my pages manually as the Google crawler renders Angular sites itself now

score 8 · Answer 1 · answered Jan 07 '14 at 15:53

8

I had the same problem but could not use explicit bootstrapping because angular e2e tests don't work with explicit bootstrap.

I solved the problem by using

<html id="ng-app" class="ng-app: appmodule;">

instead of

<html ng-app="appmodule">

htmlunit tests work and e2e tests work as well.

Very likely, htmlunit doesn't (fully?) support document.querySelectorAll(). This method is used by angularInit() to find ng-app directives.

The syntactic variant for the ng-app directive works around the document.querySelectorAll() calls in angularInit().

answered Jan 07 '14 at 15:53

stephanme

329
2
4

This should be the accepted answer, as it solved the problem where HTMLUnit was not resolving the angular directives. Thanks! – chaitanya Apr 16 '14 at 18:47
This seems to help, due to HtmlUnit's poor support for AngularJS, but it doesn't resolve all of the issues. Hopefully 2.15 will be better... – Splaktar Apr 19 '14 at 05:20

score 1 · Answer 2 · answered Dec 26 '13 at 14:00

I had same problem with "unevaluated angular expressions" if I use HtmlUnit. The solution is to bootstrap application manually. Reproduction steps:

Minimal example of app working in browser, but not with HtmlUnit:

<!doctype html>
<html ng-app>
<head>
    <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.2.6/angular.min.js"></script>
</head>
<body>
    <div>
        <label>Name:</label> <input type="text" ng-model="yourName"
            placeholder="Enter a name here">
        <hr>
        <h1>Hello {{yourName}}!</h1>
    </div>
</body>
</html>

Modification steps:

Bootstrap manually
Remove ng-app to not bootstrap app twice
If you use $http or like you should re-sync it with:

webClient.setAjaxController(new NicelyResynchronizingAjaxController());

And now working example:

<!doctype html>
<html>
<head>
    <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.2.6/angular.min.js"></script>
    <script>
        angular.element(document).ready(function() {
            angular.module('myApp', []);
            angular.bootstrap(document, ['myApp']);
        });
    </script>
</head>
<body>
    <div>
        <label>Name:</label> <input type="text" ng-model="yourName"
            placeholder="Enter a name here">
        <hr>
        <h1>Hello {{yourName}}!</h1>
    </div>
</body>
</html>

Test:

WebClient webClient = new WebClient();
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
HtmlPage page = webClient.getPage("http://localhost:8080/index.html");

// Initial state
assertEquals("Hello !", page.getElementsByTagName("h1").get(0).asText());

// Set value
((HtmlInput)page.getElementsByTagName("input").get(0)).setValueAttribute("world");

// New state
assertEquals("Hello world!", page.getElementsByTagName("h1").get(0).asText());

It's working solution, but not really pleasure solution. I don't know it is problem of HtmlUnit or Angularjs.

score 1 · Answer 3 · answered Mar 21 '14 at 15:51

1

The HtmlUnit issue has now been fixed. AngularJS expressions are now evaluated correctly.

https://sourceforge.net/p/htmlunit/bugs/1559/

answered Mar 21 '14 at 15:51

Marc Guillemot

11
1

What version is this fixed in? I'm using 2.14 and using BrowserVersion.CHROME or FIREFOX_24 results in unevaluated AngularJS expressions, but using BrowserVersion.INTERNET_EXPLORER_8 does appear to evaluate those expressions. – Splaktar Apr 18 '14 at 22:30
Looks like this is not yet in a release (2.14 was released in February ). This fix was made in March and is not yet part of a release. I tried to build from the Latest SVN, but it failed: https://sourceforge.net/p/htmlunit/bugs/1596/ – Splaktar Apr 19 '14 at 05:19
I've tested this again with the latest SVN branch of 2.15 and I am still required to do the old hack mentioned in the answer by @stephanme . I am using AngularJS 1.0.8. – Splaktar Apr 19 '14 at 15:10

score 0 · Answer 4 · answered Nov 26 '13 at 01:34

A similar code I have works fine when my single page app uses angularjs 1.0.4; the only thing different I have to do was tell htmlunit to use FIREFOX_17 instead of the default IE8 in htmlunit 2.12 (Similar to the link you provided but FIREFOX_17 instead of FIREFOX_10)

final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_17);

I upgraded to angularjs 1.2 and boom my page shows up with all the angular placeholders.

Ahmed Ashour · Answer 5 · 2014-05-30T18:03:04.623

0

Thanks for reporting, fixed in SVN. Please expect HtmlUnit 2.15 very soon.

The test case now works with Chrome simulation, the reason was querySelectorAll() should be defined in document/element.

Please note, it seems others have already identified the root cause, and giving a minimal test case to HtmlUnit team can have it fixed in a very short time.

Thanks again for your feedback.

edited May 30 '14 at 18:03

answered May 30 '14 at 17:51

Ahmed Ashour

5,179
10
35
56

HTMLUnit not working with AngularJS

5 Answers5