How to extract data from ajax/javascript websites using htmlunit? I m trying to extract shipment history

Question

I m trying to extract shipment history from this page http://www.aramex.com/express/track-results.aspx?q=aWQ9MzU2NDQ4MTQ3Jg%3d%3d-ULINyZQtKrw%3d.

This my code:

public void aramexTracking() {
    WebClient webClient  = new WebClient(BrowserVersion.CHROME);
    String trackingId = "9181468833";       
    HtmlPage page1, page2;

            try {

            page1 = webClient.getPage("http://www.aramex.com/express/track.aspx");


                                     webClient.getOptions().setThrowExceptionOnScriptError(false);

                               webClient.getOptions().setPrintContentOnFailingStatusCode(false);


           webClient.setCssErrorHandler(new com.gargoylesoftware.htmlunit.SilentCssErrorHandler());



                //Submitting form on Tracking Page
                HtmlForm form = page1.getFormByName("aspnetForm");

                HtmlButtonInput button =  form.getInputByName("ctl00$ctl00$MainContent$InnerMainContent$btnGo");

                HtmlTextArea textArea = form.getTextAreaByName("ShipmentNumber");
                textArea.setText(trackingId);

                page2 = button.click();

                List<?> list = page2.getByXPath("//div[@id='dvSearchResults']/text()");



            } catch (FailingHttpStatusCodeException | IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }       
}

Answer 1

Please, post a valid tracking number. I tried random one - 3974937493 and want to suggest another xpath:

HtmlTable table = (HtmlTable) page2.getFirstByXPath("//div[@id='MainContent']//table//table");

After that, parse rows of the table as usual

if (table.getCellAt(1,0) != null) System.out.println(table.getCellAt(1,0).asText();