How to get data from the Java web scraping API?

Question

I am trying to get table data from the following url:

and I wrote this code with the help of jaunt API

package org.open.browser;

import com.jaunt.Element;
import com.jaunt.Elements;
import com.jaunt.JauntException;
import com.jaunt.UserAgent;

public class ICICIScraperDemo {

    public static void main(String ar[]) throws JauntException{
        
        UserAgent userAgent = new UserAgent();         //create new userAgent (headless browser)
        userAgent.visit("https://www.icicidirect.com/idirectcontent/Research/TechnicalAnalysis.aspx/companyprofile/inftec");     
       Elements links = userAgent.doc.findEvery("<div class=expander>").findEvery("<a>");  //find search result links
        String url = null;
        for(Element link : links) {
            if(link.innerHTML().equalsIgnoreCase("Company Details")){
                  url = link.getAt("href");
            }
        }
        /*userAgent = new UserAgent(); */        //create new userAgent (headless browser)
        userAgent.visit(url);   
        System.out.println(userAgent.getSource());
        Elements results = userAgent.doc.findEvery("<tr>").findEvery("<td>");
          System.out.println(results);
    }
}

But it didn't work.

Then I tried another API called htmlunit and wrote below code

public void htmlUnitEx(){
        String START_URL = "https://www.icicidirect.com/idirectcontent/Research/TechnicalAnalysis.aspx/companyprofile/inftec";
                try {
                    WebClient webClient = new WebClient(BrowserVersion.CHROME);
                    HtmlPage page = webClient.getPage(START_URL);
                    WebResponse webres = page.getWebResponse();
                    //List<HtmlAnchor> companyInfo = (List) page.getByXPath("//input[@id='txtStockCode']");
                     HtmlTable companyInfo = (HtmlTable) page.getFirstByXPath("//table");
                        for(HtmlTableRow  item : companyInfo.getBodies().get(0).getRows()){  
                            String label = item.getCell(1).asText();
                            System.out.println(label);
                             if(!label.contains("Registered Office")){
                                    continue ;
                                }
}
}

But this also not giving the result .

Can someone please help how to get the data from the above url and other Anchor url in a single session?

What do you mean "didn't work"? The `` tags are all JavaScript. What is it you're trying to get? The real data loads in an iframe - use your browser tools to see the call to, for example, `https://www.icicidirect.com/idirectcontent/basemasterpage/ContentDataHandler.ashx?icicicode=INFTEC` — stdunbar, Sep 06 '18 at 02:14
@stdunbar Thanks for your comment ,do you mean the url for table data creation is different than url i posted ? Also these API wont work to get data from the Page? And if you saw this url when loading right hand side one table with data ,that data i want to store. — Subodh Joshi, Sep 06 '18 at 02:17

score 0 · Answer 1 · answered Sep 07 '18 at 18:31

Using HtmlUnit you can do this

    String url = "https://www.icicidirect.com/idirectcontent/Research/TechnicalAnalysis.aspx/companyprofile/inftec";

    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_60)) {
        HtmlPage page = webClient.getPage(url);
        webClient.waitForBackgroundJavaScript(1000);

        final DomNodeList<DomNode> divs = page.querySelectorAll("div.bigcoll");
        System.out.println(divs.get(1).asText());
    }

Two things to mention:

you have to wait after the getPage call a bit because some parts are created by javascript/AJAX
there are many way to find elements on a page (see Finding a specific element). I have done only a quick hack to show the code is working.

How to get data from the Java web scraping API?

1 Answers1