4

I want to split a file into multiple chunks (in this case, trying lengths of 300) and base64 encode it, since loading the entire file to memory gives a negative array exception when base64 encoding it. I tried using the following code:

int offset = 0;
bis = new BufferedInputStream(new FileInputStream(f));
while(offset + 300 <= f.length()){
    byte[] temp = new byte[300];
    bis.skip(offset);
    bis.read(temp, 0, 300);
    offset += 300;
    System.out.println(Base64.encode(temp));
}
if(offset < f.length()){
    byte[] temp = new byte[(int) f.length() - offset];
    bis.skip(offset);
    bis.read(temp, 0, temp.length);
    System.out.println(Base64.encode(temp));
}

At first it appears to be working, however, at one point it switches to just printing out "AAAAAAAAA" and fills up the entire console with it, and the new file is corrupted when decoded. What could be causing this error?

Adrian Veliz
  • 124
  • 1
  • 7
  • your offset is not set correctly int count = bis.read(temp, 0, 300); offset += count; – PWC Aug 14 '15 at 02:17

2 Answers2

2

skip() "Skips over and discards n bytes of data from the input stream", and read() returns "the number of bytes read".

So, you read some bytes, skip some bytes, read some more, skip, .... eventually reaching EOF at which point read() returns -1, but you ignore that and use the content of temp which contains all 0's, that are then encoded to all A's.

Your code should be:

try (InputStream in = new BufferedInputStream(new FileInputStream(f))) {
    int len;
    byte[] temp = new byte[300];
    while ((len = in.read(temp)) > 0)
        System.out.println(Base64.encode(temp, 0, len));
}

This code reuses the single buffer allocated before the loop, so it will also cause much less garbage collection than your code.

If Base64.encode doesn't have a 3 parameter version, do this:

try (InputStream in = new BufferedInputStream(new FileInputStream(f))) {
    int len;
    byte[] temp = new byte[300];
    while ((len = in.read(temp)) > 0) {
        byte[] data;
        if (len == temp.length)
            data = temp;
        else {
            data = new byte[len];
            System.arraycopy(temp, 0, data, 0, len);
        }
        System.out.println(Base64.encode(data));
    }
}
Andreas
  • 154,647
  • 11
  • 152
  • 247
0

Be sure to use a buffer size that is a multiple of 3 for encoding and a multiple of 4 for decoding when using chunks of data.

300 fulfills both, so that is already OK. Just as an info for those trying different buffer sizes.

Keep in mind, that reading from a stream into a buffer can in some cicumstances result in a buffer not being fully filled, even though the end of the stream was not yet reached. Might be possible when reading from an internet stream and a timeout occures. You can heal that, but taking that into account would lead to much more complex coding, that would not be educational anymore.

Quippy
  • 1