Apache Tika: Parsing a text file omits last part?

Question

I am trying to parse a plain text file using Tika but getting inconsistent behavior.

More specifically, I have defined a simple handler as follows:

public class MyHandler extends DefaultHandler
{
     @Override
     public void characters(char ch[], int start, int length) throws SAXException
     {
        System.out.println(new String(ch));
     }
}

Then, I parse the file ("myfile.txt") as follows:

Tika tika = new Tika();
InputStream is = new FileInputStream("myfile.txt");

Metadata metadata = new Metadata();
ContentHandler handler = new MyHandler();

Parser parser = new TXTParser();
ParseContext context = new ParseContext();

String mimeType = tika.detect(is);
metadata.set(HttpHeaders.CONTENT_TYPE, mimeType);

tikaParser.parse(is, handler, metadata, context);

I would expect all the text in the file to be printed out on screen, but a small part in the end is not. More specifically, the characters() callback keeps reading 4,096 characters per callback but in the end it apparently leaves out the last 5,083 characters of this particular file (which is a few MB long), so it even goes beyond missing the last callback.

Also, testing on another, small file, which is about 5,000 characters long, no callback seems to take place!

The MIME type is correctly detected as text/plain in both cases.

Any ideas?

Thanks!

Are you able to put together a small JUnit unit test that shows the problem? There are several unit tests for Text parsing in Tika which all parse, so clearly another one is needed for your case! — Gagravarr, Jul 08 '11 at 19:40

score 4 · Accepted Answer · answered Jul 07 '11 at 20:49

What version of Tika are you using? Looking at the source code it reads chunks of 4096 bytes which can be seen on line 129 of TXTParser. At line 132 the characters(...) routine is invoked.

In short, the target code is:

   char[] buffer = new char[4096];
   int n = reader.read(buffer);
   while (n != -1) {
       xhtml.characters(buffer, 0, n);
       n = reader.read(buffer);
   }

where reader is a BufferedReader. I cannot see any flaw in this code, hence I'm thinking you might be working an older version?

I can't see any flaw with the above code, either, but the issue remains. Very strange! — PNS, Jul 24 '11 at 12:23

Apache Tika: Parsing a text file omits last part?

1 Answers1