Unable to retrieve web data in java using tidy and Xpath

Question

What I am trying to do is scrape a simple inner HTML from a XHTML file. I have narrowed down my search to the element node, but I fail to retrieve the information.

PLEASE NOTE: the element node has no child node. I get a null pointer exception for doing that

here is the HTML SNIPPET

    <div id="dvTitle" class="titlebtmbrdr01" style="line-height: 22px;">BAJAJ AUTO LTD.       </div>

PLease also NOTE that this file has namespace as http://www.w3.org/1999/xhtml

You can see that I have the div element from which I want BAJAJ AUTO LTD.

Here is the code that i am using

    import java.io.IOException;
     import java.net.MalformedURLException; 
      import java.net.URL;
      import java.util.Vector;

    import javax.xml.xpath.XPath;
    import javax.xml.xpath.XPathConstants;
    import javax.xml.xpath.XPathExpression;
      import javax.xml.xpath.XPathExpressionException;
    import javax.xml.xpath.XPathFactory;

    import jxl.read.biff.BiffException;
    import jxl.write.WriteException;
    import jxl.write.biff.RowsExceededException;

    import org.w3c.dom.Document;
    import org.w3c.dom.Element;
      import org.w3c.dom.Node;
      import org.w3c.dom.NodeList;
    import org.w3c.dom.Text;

    import com.sun.org.apache.xml.internal.serialize.Serializer;


    public class BSEQuotesExtractor implements valueExtractor {

@Override
public Vector<String> getName(Document d) throws XPathExpressionException,            RowsExceededException, BiffException, WriteException, IOException {
    // TODO Auto-generated method stub
    XPathFactory factory = XPathFactory.newInstance();
    XPath xpath = factory.newXPath();
    xpath.setNamespaceContext(new MynamespaceContext());


    Object result = xpath.evaluate("//*[@id='dvTitle']",d, XPathConstants.NODESET);
    NodeList nodes = (NodeList) result;

    System.out.println(nodes.getLength());
    System.out.println(nodes.item(0).getNodeName());
    System.out.println(nodes.item(0).getAttributes().item(1).getNodeName());
    System.out.println(nodes.item(0).getAttributes().item(1).getNodeValue());
    System.out.println(nodes.item(0).getTextContent());

    return null;
}

public static void main(String[] args) throws MalformedURLException, IOException, XPathExpressionException, RowsExceededException, BiffException, WriteException{
    BSEQuotesExtractor q = new BSEQuotesExtractor();
    DOMParser parser = new DOMParser(new URL("http://www.bseindia.com/bseplus/StockReach/StockQuote/Equity/BAJAJ%20AUTO%20LTD/BAJAJAUT/532977/Scrips").openStream());
    Document d = parser.getDocument();
    q.getName(d);

}

        }

And this is the output I get

1
div
dvTitle
null

Now why do I get that null? I should get BAJAJ AUTO LTD.

Please reformat your question to make it more readable. – MvG Jul 11 '12 at 12:09 — MvG, Jul 11 '12 at 12:09

score 1 · Accepted Answer · answered Jul 11 '12 at 12:08

1

When I open the page your code references, that div actually is empty for me:

<div class="titlebtmbrdr01" id="dvTitle" style="line-height: 22px;"></div>

So perhaps you should save the page content to some file to examine if it is the same for you. If it is, but your browser displays things differently, then figure out what combination of cookies and other headers makes a difference there.

answered Jul 11 '12 at 12:08

MvG

57,380
22
148
276

yes you are right!!!I used fire bug and there it showed that BAJAJ AUTO LTD. in between those div elements. but when i open html page i seeit to be empety , so what do you suggest??how do i scrape that data from that website. I have climbed one step of ladder from your help please help me out now. – user1511443 Jul 12 '12 at 19:25
Perhsps go for ` – MvG Jul 12 '12 at 20:15

Unable to retrieve web data in java using tidy and Xpath

1 Answers1