Java.util.zip.DataFormatException: incorrect header check

Question

First post, usually I find what Im looking for in other threads but not this time:

Im using javas Deflater and Inflater to compress/ decompress some data I send between a server and client application that Im working on.

It works just fine for 99% of my tests. However there is one particular dataset that when inflated throws this exception from the inflater.inflate() method:

DataFormatException: incorrect header check

There is nothing special about the data compared to the other runs. Its just a bunch of numbers seperated by commas "encoded" as a String and then done .getBytes() to. The only thing I know is that its a bit bigger this time. There is not encoding happening anywhere between the compression -> decompression steps.

This is the code to send something to either the client or the server. The code is shared.

OutputStream outputStream = new DataOutputStream(socket.getOutputStream());
byte[] uncompressed = SOMEJSON.toString().getBytes();
int realLength = uncompressed.length;

// compress data
byte[] compressedData = ByteCompression.compress(uncompressed);
int compressedLength = compressedData.length;
    
outputStream.write(ByteBuffer.allocate(Integer.BYTES).putInt(compressedLength).array());
outputStream.write(ByteBuffer.allocate(Integer.BYTES).putInt(realLength).array());
    
outputStream.write(compressedData);
outputStream.flush();

This is the code to receive data (either client or server) also shared:

DataInputStream dataIn = new DataInputStream(socket.getInputStream());
int compressedLength = dataIn.readInt();
int realLength = dataIn.readInt();

errorhandling.info("Packet Reader", "Expecting " + compressedLength + " (" + realLength + ") bytes.");
byte[] compressedData = new byte[compressedLength];
            
int readBytes = 0;
while (readBytes < compressedLength) {
    int newByteAmount = dataIn.read(compressedData);
                
    // catch nothing being read or end of line
    if (newByteAmount <= 0) {
        break;
    }
    readBytes += newByteAmount;
}
            
if (readBytes != compressedLength) {
    errorhandling.info("Packet Reader", "Read byte amount differs from expected bytes.");
    return new ErrorPacket("Read byte amount differs from expected bytes.").create();
}

byte[] uncompressedData = ByteCompression.decompress(compressedData, realLength);
String packetData = new String(uncompressedData);

Here are the methods to compress and decompress a byteArray (you guessed right its shared):

public static byte[] compress(byte[] uncompressed) {
        Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION);
        deflater.setInput(uncompressed);
        deflater.finish();

        byte[] compressed = new byte[uncompressed.length];
        int compressedSize = 0;
        while (!deflater.finished()) {
            compressedSize += deflater.deflate(compressed);
        }
        
        deflater.end();

        return Arrays.copyOfRange(compressed, 0, compressedSize);
    }
    
    public static byte[] decompress(byte[] compressed, int realLength) throws DataFormatException {     
        Inflater inflater = new Inflater(true);
        inflater.setInput(compressed);

        byte[] uncompressed = new byte[realLength];
        while (!inflater.finished()) {
            inflater.inflate(uncompressed); // throws DataFormatException: incorrect header check (but only super rarely)
        }
        inflater.end();

        return uncompressed;
    }

So far Ive tried differnt compression levels and messing with the "nowrap" option for both Deflater and Inflater (all combinations):

// [...]
Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION, true);
// [...]
Inflater inflater = new Inflater(true);

But that just results in these exceptions (but again only for that one particulat dataset):

DataFormatException: invalid stored block lengths
DataFormatException: invalid distance code

Im sorry for this wall of text but at this point I really dont know anymore what could be causing this issue.

What are first few bytes of the data you are trying to inflate that fails? — Mark Adler, Jun 16 '21 at 01:21
Ive now found out that the input data is indeed wrong. Writing the data to a file FROM THE CLIENT before inflating shows that the header is incorrect. Interestingly if I save the data to a file SERVERSIDE and then read CLIENTSIDE from that same file the inflation works just fine. So for whatever reason the data is being mangled between server and client somewhere when sending it over. But only for data bigger than other data that works fine. — JakobSailer, Jun 16 '21 at 20:01
first bytes from broken data: 43 12 56 19 first bytes from good data: 78 DA 7D 9C — JakobSailer, Jun 16 '21 at 20:17
You're using every API under the sun to send and receive the integers when you could just use `DataOutputStream.writeInt()` and `DataInputStream.readInt()`. Don't overcook this. And similarly you could get rid of most of the rest of this and use `Inflater/Deflater` input and output streams. — user207421, Jun 17 '21 at 00:39

JakobSailer · Accepted Answer · 2021-06-17T07:11:39.720

0

Alright here is the solution:

My assumption was that this loop would APPEND new read data to the byte array where it last stopped THIS IS NOT THE CASE (it seems to stop reading after 2^16 bytes so thats why I dont get this issue with smaller packets).

This is wrong:

int readBytes = 0;
while (readBytes < compressedLength) {
    int newByteAmount = dataIn.read(compressedData); // focus here!
    readBytes += newByteAmount;
}

So whats happening is that the data is read correctly however the output array is overwriting itself!! Thats why I see wrong data at the start and a bunch of 00 00 at the end (because it never actually reached that part of the array)!

Using this instead fixed my issue:

dataIn.readFully(compressedData);

What concerns me is that I see the first variant of the code A LOT. Thats what I found when googling it.

edited Jun 17 '21 at 07:11

answered Jun 16 '21 at 20:49

JakobSailer

15
1
4

Hopefully you will accept this answer when you can. Unless, of course, somebody writes a better one, but you seemed to have covered all the basics. – NomadMaker Jun 16 '21 at 21:08
'Overriding'? Do you mean 'overwriting'? If you mean that `read(byte[] buffer)` writes into `buffer` starting at the 0th element, that's what it's supposed to do. Your code relied on false assumptions. – user207421 Jun 17 '21 at 00:34
Overwriting is was I meant. I edited the awnser. Its easy to spot with hindsight that my assumtion was incorrect yes. – JakobSailer Jun 17 '21 at 07:15

Java.util.zip.DataFormatException: incorrect header check

1 Answers1