How to send compressed data using UDP or TCP (lua / java)

Question

I am making a server with lua clients and Java server. I need some data to be compressed in order to reduce the data flow.

In order to do this I use LibDeflate for compressing the data on the client side

local config = {level = 1}
local compressed = LibDeflate:CompressDeflate(data, config)
UDP.send("21107"..compressed..serverVehicleID) -- Send data

On the server I use this to receive the packet (TCP)

out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedReader(new 
InputStreamReader(clientSocket.getInputStream(), "UTF-8"));
String inputLine;

while ((inputLine = in.readLine()) != null) { // Wait for data
    Log.debug(inputLine); // It is what get printed in the exemple
    String[] processedInput = processInput(inputLine);
    onDataReceived(processedInput);
}

I already tried sending it using UDP and TCP, the problem is the same. I tried using LibDeflate:CompressDeflate and LibDeflate:CompressZlib I tried tweaking the config Nothing works :/

I expect to receive one packet with the whole string But I received few packets each of them contains compressed characters. exemple (each line is the server think that he receive a new packet):
_{(source: noelshack.com)}

You need to uncompress the data on the Java side. Instead, it looks like you are trying to read the compressed data as if it was text. — Stephen C, Apr 10 '19 at 13:44
The problem is that I receive the data in multiple lines. The onDataReceived function is called about 10 times for 1 compressed packet so if I try to uncompress the data it doesn't work because I don't have the whole packet but just a part — jojos38, Apr 10 '19 at 13:48
Use https://docs.oracle.com/javase/8/docs/api/java/util/zip/DeflaterInputStream.html. Obviously this won't work with UDP. For that you will need to carefully assemble the packets into a byte array, then use a ByteArrayInputStream wrapped in a DeflaterInputStream — Stephen C, Apr 10 '19 at 13:51
If you're not sending lines, why are you trying to receive them? — David Schwartz, Apr 13 '19 at 22:30

jojos38 · Answer 1 · 2019-04-13T22:12:29.613

After a lot of research I finnaly managed to fix my problem ! I used this :

DataInputStream in = new DataInputStream(new BufferedInputStream(clientSocket.getInputStream()));

int count;
byte[] buffer = new byte[8192]; // or 4096, or more

while ((count = in.read(buffer)) > 0) {
    String data = new String(buffer, 0, count);
    Do something...
}

I still haven't tested to see if the received compressed string works, I'll update my post when I try out.

EDIT: It seems to work

The only problem now is that I don't know what to do when the packet is bigger than the buffer size. I want to have something that work in every situation and since some packet are bigger than 8192 they are just cut in half.

score 0 · Answer 2 · answered Apr 10 '19 at 14:11

0

Assuming that the client side sends a single compressed "document", your server-side code should look something like this (TCP version):

is = new DeflaterInputStream(clientSocket.getInputStream());
in = new BufferedReader(new InputStreamReader(is, "UTF-8"));
String inputLine;

while ((inputLine = in.readLine()) != null) { 
    ...
}

The above is untested, and also needs exception handling and code to ensure that the streams always get closed.

The trick is that your input pipeline needs to decompress the data stream before you attempt to read / process it as lines of text.

answered Apr 10 '19 at 14:11

Stephen C

698,415
94
811
1,216

Thanks for your help, I just tried, unfortunatly same problem, but I wonder if it's not lua that send the data in multiple packets ?? – jojos38 Apr 10 '19 at 15:31
Edit of my other comment : No it's not data that send the data in multiple packet I think because when I send the data uncompressed it work fine. Howether I noticed that the server and the client are both stuck until I disconnect the client from the server. Once I do that the server receive the compressed data like on the image on the post. – jojos38 Apr 10 '19 at 15:50

How to send compressed data using UDP or TCP (lua / java)

2 Answers2