Transmiting/receiving compressed data with sockets: how to properly receive the data sent from the client

Question

I have developed a client-server chat using the Sockets and it works great, but when I try to transmit data with Deflate compression it doesn't work: the output is "empty" (actually it's not empty, but I'll explain below).

The compression/decompression part is 100% working (I have already tested it), so the problem must be elsewhere in the transmission/receiving part.

I send the message from the client to the server using these methods:

// streamOut is an instance of DataOutputStream
// message is a String

if (zip) { // zip is a boolean variable: true means that compression is active
    streamOut.write(Zip.compress(message)); // Zip.compress(String) returns a byte[] array of the compressed "message"
} else {
    // if compression isn't active, the client sends the not compressed message to the server (and this works great)
    streamOut.writeUTF(message);
}
streamOut.flush();

And I receive the message from the client to the server using these other methods:

// streamIn is an instace of DataInputStream

if (server.zip) { // same as before: true = compression is active
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    byte[] buf = new byte[512];
    int n;

    while ((n = streamIn.read(buf)) > 0) {
        bos.write(buf, 0, n);
    }

    byte[] output = bos.toByteArray();
    System.out.println("output: " + Zip.decompress(output)); // Zip.decompress(byte[]) returns a String of decompressed byte[] array received
} else {
    System.out.println("output: " + streamIn.readUTF()); // this works great
}

Debugging a little bit my program, I've discovered that the while loop never ends, so:

byte[] output = bos.toByteArray();
System.out.println("output: " + Zip.decompress(output));

is never called.

If I put those 2 lines of code in the while loop (after bos.write()), then all works fine (it prints the message sent from the client)! But I don't think that's the solution, because the byte[] array received may vary in size. Because of this I assumed that the problem is in the receiving part (the client is actually able to send data).

So my problem became the while loop in the receiving part. I tried with:

while ((n = streamIn.read(buf)) != -1) {

and even with the condition != 0, but it's the same as before: the loop never ends, so the output part is never called.

score 2 · Answer 1 · answered Jul 25 '12 at 18:51

-1 is only returned when the socket is closed or broken. You could close the socket after sending your zipped content, and your code would start working. But I suspect you want to keep the socket open for more (future) chat messages. So you need some other way of letting the client know when a discrete message has been fully transmitted. Like Patrick suggested, you could transmit the message length before each zipped payload.

You might be able to leverage something in the deflate format itself, though. I think it has a last-block-in-stream marker. If you're using java.util.zip.Inflater have a look at Inflater.finished().

Thanks for the advice! Now my Zip.compress() method adds automatically 4 bytes at the start of the output byte[] array. These 4 bytes represents the length of the compressed message sent. — HBv6, Jul 26 '12 at 08:39

Patrick · Accepted Answer · 2012-07-25T18:36:10.760

The read function will not return a -1 until the stream is closed. What you can do is calculate the number of bytes that should be sent from the server to the client, and then read that number of bytes on the client side.

Calculating the number of bytes is as easy as sending the length of the byte array returned from the Zip.compress function before the actual message, and then use the readInt function to get that number.

Using this algorithm makes sure that you read the correct number of bytes before decompressing, so even if the client actually reads 0 bytes it will continue to read until it receives all bytes it wants. You can do a streamIn.read(buf, 0, Math.min(bytesLeft, buf.length)) to only read as many bytes you want.

score 0 · Answer 3 · answered Jul 25 '12 at 19:54

Your problem is the way you are working with stream. You must send some meta-data so your client know what to expect as data. Idealy you are creating a protocol/state machine to read the stream. For your example, as a quick and dirt solution, send something like data size or a termination sequence or something.

Example of solution:
Server: send the "data size" before the compressed data
Client: wait for the "data size" bytes. Now loop till read is equal or greater "data size" value. Something like:

while( streamIn.ready() && dataRead < dataExpected)
{
    dataRead += streamIn.read(buf);
}

Of course you need to read the dataExpected before, with a similar code.

Tip: You could also use UDP if you dont mind having the possibility to lose data. Its easier to program with datagrams...

Transmiting/receiving compressed data with sockets: how to properly receive the data sent from the client

3 Answers3