Scraping our site to check for Javascript errors

Question

I am kind of desperate here. For a couple of days I have been trying to create a web scraper that can go through our website and check for Javascript errors.

The big problem is that I only know Java and it seems that GhostDriver isn't maintained anymore. The guy from Ghostdriver refers to JbrowserDriver. JBrowserDriver however has no option to collect the Javascript errors from the console.

Then I tried HTMLUnit which is way to eager on throwing errors, not javascript related. So after fiddling with it half a day, I threw in the towel on HtmlUnit.

I could revert to plain old WebDriver but that would involve too much boilerplate.

Does anyone of you guys have any suggestions?

try browsermobproxy with selenium and that should give you everything — Mrunal Gosar, Apr 12 '16 at 13:01

score 1 · Accepted Answer · answered Apr 12 '16 at 13:19

1

You could inject some javascript into the head section of each window to log all errors into a hidden div. Selenium could then get the text from this div and parse it into a report of all errors that occurred on the page.

For example, given the following page layout:

<html>
    <head>
        <script>
            window.onerror = function(e) {
                document.getElementById("hidden-selenium-log").innerText += e.toString() + ";";
            }
        </script>
    </head>
    <body>

        <div id="hidden-selenium-log" style="display: none;">
        </div>

        <div id="broken-button" onclick="unknownFunction()">broken</div>

    </body>
</html>

The script in the head tag would write all javascript errors into the div hidden-selenium-log. Clicking on the div broken-button would trigger the error event handler and log it into the hidden selenium log.

After interacting with the page, you could then do something simple like:

Driver.FindElement(By.Id("hidden-selenium-log")).text.split(";");

This would get the text in the hidden selenium log and then split it by the semi-colon, a character I appended after each error logged.

answered Apr 12 '16 at 13:19

Ben C

476
5
16

I am currently trying such approach with JBrowserDriver. You can inject javascript in the head and then call it when the page is rendered. So far without success. – Homewrecker Apr 12 '16 at 13:22
What kind of issues are you encountering with this approach? Does the javascript just not get called? – Ben C Apr 12 '16 at 13:37
I ibsert and then I perform: driver.executeScript("return window.jsErrors"); But I get back null and I am sure there is a Javascript error on the page. – Homewrecker Apr 12 '16 at 14:08
Maybe add in another script on the page that causes an error just to make sure? – Ben C Apr 12 '16 at 14:24
Where do you insert your script? It could be that the error has happened before your event is registered. – Ben C Apr 12 '16 at 14:32
I do it using the jbrowserdriver headscript function, so it should be inserted before the page get called – Homewrecker Apr 12 '16 at 15:35

score 0 · Answer 2 · answered May 21 '16 at 11:31

Logging of javascript is disabled by default. You can enable it via the settings builder.

Settings settings = Settings.builder()
        .logJavascript(true)
        .build();
JBrowserDriver jBrowserDriver = new JBrowserDriver(settings);

jBrowserDriver.get("http://example.com");

Logs logs = jBrowserDriver.manage().logs();
LogEntries logEntries = logs.get("javascript");
logEntries.forEach(System.out::println);

Scraping our site to check for Javascript errors

2 Answers2