java GZIPInputStream and GZIPOutputStream not working as expected

Question

I am trying to write a series of Long's to a GZIPOutputStream, hoping to decompress the numbers later.
The following program works fine when I try it with a small number of Long's, but it will throw an exception with many Long's, like (10240).

Please see and run this code:

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.nio.ByteBuffer;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class GzipTest {
    private static final String filename = "data.gz";

    public static void main(String... args) throws Exception {
        test(32);  // OK
        test(10240);  // Why won't this work? (unexpected read: 6)
    }

    public static void test(int max) throws Exception {
        System.out.println("testing with " + max);
        FileOutputStream fos = new FileOutputStream(filename);
        GZIPOutputStream gos = new GZIPOutputStream(fos, 4096);
        long num = 10241000L;
        for (int i = 0; i < max; ++i) {
            gos.write(long2bytes(num + i));
        }
        gos.close();
        fos.close();

        GZIPInputStream gis = new GZIPInputStream(new FileInputStream(filename), 4096);
        byte[] buf = new byte[Long.BYTES];
        for (int i = 0; i < max; ++i) {
            int nread = gis.read(buf);
            if (nread != Long.BYTES) {
                throw new RuntimeException("unexpected read: " + nread);
            }
            long value = bytes2long(buf);
            if (value != num + i) {
                throw new RuntimeException("unexpected value: " + value);
            }
        }
        gis.close();
    }

    static byte[] long2bytes(long num) {
        ByteBuffer buf = ByteBuffer.allocate(Long.BYTES);
        buf.putLong(num);
        return buf.array();
    }

    static long bytes2long(byte[] bytes) {
        return ByteBuffer.wrap(bytes).getLong();
    }
}

The outputs:

testing with 32
testing with 10240
Exception in thread "main" java.lang.RuntimeException: unexpected read: 6
    at GzipTest.test(GzipTest.java:31)
    at GzipTest.main(GzipTest.java:12)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

What is the problem you are having? Note: read is only guaranteed to read 1 byte (or more) Why not use DataOutputStream and writeLong/readLong or use readFully to always read 8 bytes. — Peter Lawrey, Nov 01 '16 at 09:22
@PeterLawrey The above program works fine when I try to de/compress a few of `Long`'s, like 32, but won't work when I try many `Long`'s, like 10240. — adamsmith, Nov 01 '16 at 09:25
What does "won't work" mean? --- Also, in `bytes2long`, you should just use `return ByteBuffer.wrap(bytes).getLong();`. — Andreas, Nov 01 '16 at 09:34
I mean why `test(32);` passes but `test(10240);` throws an exception. — adamsmith, Nov 01 '16 at 09:35
What exception? Show us the *full* stacktrace! "won't work" is a useless description. — Andreas, Nov 01 '16 at 09:35
So you only got 6 bytes when you were expecting 8. See [comment by @PeterLawrey](http://stackoverflow.com/questions/40357123/java-gzipinputstream-and-gzipoutputstream-not-working-as-expected#comment67967654_40357123) for reason and solution. — Andreas, Nov 01 '16 at 09:39
@PeterLawrey Now I see what you mean. `read` does not guarantee full read. You are right. Thanks man! — adamsmith, Nov 01 '16 at 09:53

Robert · Accepted Answer · 2016-11-01T10:30:58.200

1

Th code works without any problems if you make the changes proposed by Peter Lawrey:

 try (DataInputStream in = new DataInputStream(new GZIPInputStream(new FileInputStream(filename), 4096))) {
        byte[] buf = new byte[Long.BYTES];
        for (int i = 0; i < max; ++i) {
            in.readFully(buf);
            long value = bytes2long(buf);
            if (value != num + i) {
                throw new RuntimeException("unexpected value: " + value);
            }
        }
    }

Or you can simplify everything to:

 try (DataInputStream in = new DataInputStream(new GZIPInputStream(new FileInputStream(filename), 4096))) {
        for (int i = 0; i < max; ++i) {
            long value = in.readLong();
            if (value != num + i) {
                throw new RuntimeException("unexpected value: " + value);
            }
        }
    }

edited Nov 01 '16 at 10:30

answered Nov 01 '16 at 09:36

Robert

39,162
17
99
152

I fear there might be some overhead with DataInputStream. Besides, is that the only way to work with `GZIPInputStream`? – adamsmith Nov 01 '16 at 09:43
The overhead with DataInputStream is less that your way to read/write a long using a new ByteBuffer for each long. DataInputStream offers the nice method `readLong()` with does everything for you. Therefore you only need DataInputStream and DataOutputStream for reading and writing long values. – Robert Nov 01 '16 at 10:29

java GZIPInputStream and GZIPOutputStream not working as expected

1 Answers1