0

I am using HtmlUnit in a Java application to load a web page and submit its form. I get the form and the submit input without any problems, but when I call the function click() on that element, it triggers a Cast Exception and doesn't seem to execute the click event. Here's my webClient configuration: 

this.webClient = new WebClient(BrowserVersion.CHROME);

webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(true);
webClient.getOptions().setCssEnabled(false);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setRedirectEnabled(true);

final HtmlPage jobPage = webClient.getPage("https://mySite/j#edit-job?id="+job.getJobID());
webClient.waitForBackgroundJavaScript(15000);

Here's the code of the page loading and input selection:

HtmlForm form = jobPage.getFormByName("");
HtmlSubmitInput submit = null;
final List<HtmlInput> inputs = form.getInputsByName("");

for(HtmlInput input : inputs){

    if(input.getAttribute("id").equals("postFormSubmit")){
        submit = (HtmlSubmitInput)input;
        break;
    }
}


if(submit != null){
    try {
        HtmlPage confirmUpdate = submit.click();
        webClient.waitForBackgroundJavaScript(15000);

        System.out.println("response : \n\n");
        System.out.println(confirmUpdate.asText());
        System.out.println("end response");

    }
    catch(Exception e){
        e.printStackTrace();
    }
}
else {
    System.out.println("submit input non trouvé");
}

I also tried to execute some javascript manually by doing: 

jobPage.executeJavascript("javascript:document.getElementById('postFormSubmit').click();");

But I get the same exception:

INFOS: Caught script exception
======= EXCEPTION START ========
Exception class=[java.lang.ClassCastException]
com.gargoylesoftware.htmlunit.ScriptException: com.gargoylesoftware.htmlunit.javascript.host.html.HTMLInputElement cannot be cast to com.gargoylesoftware.htmlunit.html.HtmlElement
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:898)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:637)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:518)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:827)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:799)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScriptFunctionIfPossible(HtmlPage.java:2470)
    at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeEventListeners(EventListenersContainer.java:259)
    at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeBubblingListeners(EventListenersContainer.java:325)
    at com.gargoylesoftware.htmlunit.javascript.host.event.EventTarget.fireEvent(EventTarget.java:188)
    at com.gargoylesoftware.htmlunit.html.DomElement$2.run(DomElement.java:1389)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:637)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:518)
    at com.gargoylesoftware.htmlunit.html.DomElement.fireEvent(DomElement.java:1394)
    at com.gargoylesoftware.htmlunit.html.DomElement.fireEvent(DomElement.java:1362)
    at com.gargoylesoftware.htmlunit.html.HtmlForm.submit(HtmlForm.java:116)
    at com.gargoylesoftware.htmlunit.html.HtmlSubmitInput.doClickStateUpdate(HtmlSubmitInput.java:90)
    at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:999)
    at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:944)
    at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:884)
    at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:865)
    at model.IndeedManip$2.run(IndeedManip.java:309)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassCastException: com.gargoylesoftware.htmlunit.javascript.host.html.HTMLInputElement cannot be cast to com.gargoylesoftware.htmlunit.html.HtmlElement
    at com.gargoylesoftware.htmlunit.javascript.host.dom.AbstractList.addElementIds(AbstractList.java:541)
    at com.gargoylesoftware.htmlunit.javascript.host.dom.AbstractList.getIds(AbstractList.java:523)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.enumChangeObject(ScriptRuntime.java:2235)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.enumInit(ScriptRuntime.java:2136)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1740)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.NativeArray.iterativeMethod(NativeArray.java:1694)
    at net.sourceforge.htmlunit.corejs.javascript.NativeArray.execIdCall(NativeArray.java:405)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2575)
    at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.execIdCall(BaseFunction.java:321)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1540)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.NativeArray.iterativeMethod(NativeArray.java:1694)
    at net.sourceforge.htmlunit.corejs.javascript.NativeArray.execIdCall(NativeArray.java:405)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2575)
    at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.execIdCall(BaseFunction.java:321)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1540)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:416)
    at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:292)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$4.doRun(JavaScriptEngine.java:820)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:883)
    ... 27 more
Enclosed exception: 
java.lang.ClassCastException: com.gargoylesoftware.htmlunit.javascript.host.html.HTMLInputElement cannot be cast to com.gargoylesoftware.htmlunit.html.HtmlElement
    at com.gargoylesoftware.htmlunit.javascript.host.dom.AbstractList.addElementIds(AbstractList.java:541)
    at com.gargoylesoftware.htmlunit.javascript.host.dom.AbstractList.getIds(AbstractList.java:523)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.enumChangeObject(ScriptRuntime.java:2235)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.enumInit(ScriptRuntime.java:2136)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1740)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.NativeArray.iterativeMethod(NativeArray.java:1694)
    at net.sourceforge.htmlunit.corejs.javascript.NativeArray.execIdCall(NativeArray.java:405)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2575)
    at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.execIdCall(BaseFunction.java:321)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1540)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.NativeArray.iterativeMethod(NativeArray.java:1694)
    at net.sourceforge.htmlunit.corejs.javascript.NativeArray.execIdCall(NativeArray.java:405)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2575)
    at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.execIdCall(BaseFunction.java:321)
    at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:94)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1540)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:416)
    at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:292)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$4.doRun(JavaScriptEngine.java:820)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:883)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:637)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:518)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:827)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:799)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScriptFunctionIfPossible(HtmlPage.java:2470)
    at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeEventListeners(EventListenersContainer.java:259)
    at com.gargoylesoftware.htmlunit.javascript.host.event.EventListenersContainer.executeBubblingListeners(EventListenersContainer.java:325)
    at com.gargoylesoftware.htmlunit.javascript.host.event.EventTarget.fireEvent(EventTarget.java:188)
    at com.gargoylesoftware.htmlunit.html.DomElement$2.run(DomElement.java:1389)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:637)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:518)
    at com.gargoylesoftware.htmlunit.html.DomElement.fireEvent(DomElement.java:1394)
    at com.gargoylesoftware.htmlunit.html.DomElement.fireEvent(DomElement.java:1362)
    at com.gargoylesoftware.htmlunit.html.HtmlForm.submit(HtmlForm.java:116)
    at com.gargoylesoftware.htmlunit.html.HtmlSubmitInput.doClickStateUpdate(HtmlSubmitInput.java:90)
    at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:999)
    at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:944)
    at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:884)
    at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:865)
    at model.IndeedManip$2.run(IndeedManip.java:309)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
== CALLING JAVASCRIPT ==
  function (c) {
      return a.call(b.src, b.Mh, c);
  }
======= EXCEPTION END ========

It is said it is a Casting exception but I am not doing any cast and the error is located at the line of the click() event.

Do let me know if you know another way to trigger a click event with HtmlUnit.

Ahmed Ashour
  • 5,179
  • 10
  • 35
  • 56
Simon30
  • 317
  • 3
  • 12
  • What line of code is it throwing the exception on? – CraigR8806 Apr 13 '17 at 14:19
  • This one: HtmlPage confirmUpdate = submit.click(); – Simon30 Apr 13 '17 at 14:21
  • On the page that you are using this with, how does that button submit? Does it call a Javascript function or does it submit via HTML? – CraigR8806 Apr 13 '17 at 14:22
  • Please use latest version, and provide your complete case (with URL) – Ahmed Ashour Apr 13 '17 at 14:24
  • @CraigR8806 It calls some Javascript functions with ajax calls. I can't find the functions as the JS code is minified. But I think it should work because when I run "document.getElementById('postFormSubmit').click()" in the Chrome console the form is submitted. – Simon30 Apr 13 '17 at 14:25
  • @AhmedAshour I use latest version 2.26 – Simon30 Apr 13 '17 at 14:27
  • Why not try something like this instead of the for loop: `HtmlSubmitInput button = (HtmlSubmitInput)page.getElementById("postFormSubmit");` – CraigR8806 Apr 13 '17 at 14:41
  • 1
    Please read [Under what circumstances may I add “urgent” or other similar phrases to my question, in order to obtain faster answers?](https://meta.stackoverflow.com/q/326569) - the summary is that this is not an ideal way to address volunteers, and is probably counterproductive to obtaining answers. Please refrain from adding this to your questions. – halfer Apr 13 '17 at 14:44
  • @CraigR8806 I already tried it gives me the same exception. I thought using a HtmlForm would help for its submission but you're right it's better like this – Simon30 Apr 13 '17 at 14:46
  • @halfer I do have a deadline and I am already late because I am struggling on that problem since 2 days. I've also tried to manually send a post request without success... – Simon30 Apr 13 '17 at 14:55
  • I sympathise, but we are very keen here (as per the link) that no-one asks for priority over other questions, or implies their work is more important than someone else's. Those people want an answer just as much as you do. If you are frequently subject to tight deadlines, one approach is to keep to hand a list of sources of assistance that can help you quickly. There are websites that could help here, such as TopTal and AirPair. – halfer Apr 13 '17 at 15:03
  • @AhmedAshour it looks like you are a HtmlUnit expert, please try to help me, do you know if a website can block some HtmlUnit interactions? – Simon30 Apr 13 '17 at 15:16
  • Check this answer out: http://stackoverflow.com/a/3904941/5799145 it has a link to an extension for your browser that can record your actions and generate HtmlUnit code for you – CraigR8806 Apr 13 '17 at 15:21
  • Thank you @CraigR8806 but it only works on old browser (up to firefox 7) and those browsers don't even render my page properly... – Simon30 Apr 13 '17 at 16:01

2 Answers2

0

This was just fixed by commits 14092 and 14093.

You can get latest build from here.

Test case is:

/**
 * @throws Exception on test failure
 */
@Test
@Alerts(DEFAULT = "0,1,2,3,4,5,entries,forEach,item,keys,length,values",
        IE = "0,1,2,3,4,5,item,length")
public void iterator() throws Exception {
    final String html = "<html><head><title>test</title>\n"
            + "<script>\n"
            + "  function test() {\n"
            + "    var all = [];\n"
            + "    for (var i in document.querySelectorAll('*')) {\n"
            + "      all.push(i);\n"
            + "    }\n"
            + "    all.sort(sortFunction);\n"
            + "    alert(all);\n"
            + "  }\n"
            + "  function sortFunction(s1, s2) {\n"
            + "    return s1.toLowerCase() > s2.toLowerCase() ? 1 : -1;\n"
            + "  }\n"
            + "</script>\n"
            + "</head><body onload='test()'>\n"
            + "  <div>/div>\n"
            + "</body></html>";

    loadPageWithAlerts2(html);
}
Ahmed Ashour
  • 5,179
  • 10
  • 35
  • 56
  • Thank you very much ! The excpetion is not thrown anymore, but for some reasons my form is still not submitted... When I execute the `click()` event, it only refreshes the page... Do you know where that could come from? If the website is detecting we are using java or if java can't compile every asynchronous calls? – Simon30 Apr 14 '17 at 08:38
  • By the way, I detect a message on the web page which tells me _"Loading...Server error, please try again laterSorry! It looks like you are in private browsing mode, which is not supported. Please switch your browser mode."_. Do you have an idea why? Thanks in advance – Simon30 Apr 14 '17 at 09:03
0

Thanks to Ahmed Ashour this function doesn't trigger the exception anymore, but my form has never been submitted by HtmlUnit.

Finally I switched to Selenium which is way more efficient for javascript execution. I advice any one working on a web scraping application which submits ajax powered forms to use Selenium

Simon30
  • 317
  • 3
  • 12