0

I am using boilerpipe library to analyzer news articles. There news articles contain a lot of boilerplate such as copyright information, side pane of related articles, etc. Boilerpipe removes all that information. Is it possible to return the boilerplate information? I need to analyzer and extract some stuff from copyright statement, etc.

Also, does it contains some sort of confidence for each text block as to whether it is boilerplate or not?

Thanks.

abhinavkulkarni
  • 2,284
  • 4
  • 36
  • 54

1 Answers1

1

You can get the entire text or traverse the actual text blocks by using the Document classes boilerplate provides:

final HTMLDocument htmlDoc = HTMLFetcher.fetch(new URL(url));
final TextDocument doc = new BoilerpipeSAXInput(htmlDoc.toInputSource()).getTextDocument();
// doc.getText(true, true) will give you all the text
// doc.getTextBlocks will let you traverse the document
synhershko
  • 4,472
  • 1
  • 30
  • 37