Decompressing byte[] using LZ4

Question

I am using LZ4 for compressing and decompressing a string.I have tried the following way

public class CompressionDemo {

    public static byte[] compressLZ4(LZ4Factory factory, String data) throws IOException {
        final int decompressedLength = data.getBytes().length;
        LZ4Compressor compressor = factory.fastCompressor();
        int maxCompressedLength = compressor.maxCompressedLength(decompressedLength);
        byte[] compressed = new byte[maxCompressedLength];
        compressor.compress(data.getBytes(), 0, decompressedLength, compressed, 0, maxCompressedLength);
        return compressed;

    }

    public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
        LZ4FastDecompressor decompressor = factory.fastDecompressor();
        byte[] restored = new byte[data.length];
        decompressor.decompress(data,0,restored, 0,data.length);
        return new String(restored);
    }

    public static void main(String[] args) throws IOException, DataFormatException {
        String string = "kjshfhshfashfhsakjfhksjafhkjsafhkjashfkjhfjkfhhjdshfhhjdfhdsjkfhdshfdskjfhksjdfhskjdhfkjsdhfk";
        LZ4Factory factory = LZ4Factory.fastestInstance();
        byte[] arr = compressLZ4(factory, string);
        System.out.println(arr.length);
        System.out.println(deCompressLZ4(factory, arr) + "decom");
    }
}

it is giving following excpetion

Exception in thread "main" net.jpountz.lz4.LZ4Exception: Error decoding offset 92 of input buffer

The problem here is that decompressing is working only if i pass the actual String byte[] length i.e

public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
        LZ4FastDecompressor decompressor = factory.fastDecompressor();
        byte[] restored = new byte[data.length];
        decompressor.decompress(data,0,restored, 0,"kjshfhshfashfhsakjfhksjafhkjsafhkjashfkjhfjkfhhjdshfhhjdfhdsjkfhdshfdskjfhksjdfhskjdhfkjsdhfk".getBytes().length);
        return new String(restored);
    }

It is expecting the actual string byte[] size. Can someone help me with this

Answer 1

As the compression and decompressions may happen on different machines, or the machine default character encoding is not one of the Unicode formats, one should indicate the encoding too.

For the rest it is using the actual compression and decompression lengths, and better store the size of the uncompressed data too, in plain format, so it may be extracted prior to decompressing.

public static byte[] compressLZ4(LZ4Factory factory, String data) throws IOException {
    byte[] decompressed = data.getBytes(StandardCharsets.UTF_8).length;
    LZ4Compressor compressor = factory.fastCompressor();
    int maxCompressedLength = compressor.maxCompressedLength(decompressed.length);
    byte[] compressed = new byte[4 + maxCompressedLength];
    int compressedSize = compressor.compress(decompressed, 0, decompressed.length,
                                             compressed, 4, maxCompressedLength);
    ByteBuffer.wrap(compressed).putInt(decompressed.length);
    return Arrays.copyOf(compressed, 0, 4 + compressedSize);
}

public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
    LZ4FastDecompressor decompressor = factory.fastDecompressor();
    int decrompressedLength = ByteBuffer.wrap(data).getInt();
    byte[] restored = new byte[decrompressedLength];
    decompressor.decompress(data, 4, restored, 0, decrompressedLength);
    return new String(restored, StandardCharsets.UTF_8);
}

It should be told, that String is not suited for binary data, and your compression/decompression is for text handling only. (String contains Unicode text in the form of UTF-16 two-byte chars. Conversion to binary data always involves a conversion with the encoding of the binary data. That costs in memory, speed and possible data corruption.)

Answer 2

I just faced the same error on Android and resolved it based on issue below: https://github.com/lz4/lz4-java/issues/68

In short make sure you are using the same factory for both operations (compression + decompression) and use Arrays.copyOf() as below:

  byte[] compress(final byte[] data) {
     LZ4Factory lz4Factory = LZ4Factory.safeInstance();
     LZ4Compressor fastCompressor = lz4Factory.fastCompressor();
     int maxCompressedLength = fastCompressor.maxCompressedLength(data.length);
     byte[] comp = new byte[maxCompressedLength];
     int compressedLength = fastCompressor.compress(data, 0, data.length, comp, 0, maxCompressedLength);
     return Arrays.copyOf(comp, compressedLength);
}

  byte[] decompress(final byte[] compressed) {
     LZ4Factory lz4Factory = LZ4Factory.safeInstance();
     LZ4SafeDecompressor decompressor = lz4Factory.safeDecompressor();
     byte[] decomp = new byte[compressed.length * 4];//you might need to allocate more
     decomp = decompressor.decompress(Arrays.copyOf(compressed, compressed.length), decomp.length);
     return decomp;

Hope this will help.

Answer 3

restored byte[] length is to small, you should not use compressed data.length, instead you should use data[].length * 3 or more than 3.

Answer 4

I resoved like this:

public static byte[] decompress( byte[] finalCompressedArray,String ... extInfo) {
    int len = finalCompressedArray.length * 3;
    int i = 5;
    while (i > 0) {
        try {
            return decompress(finalCompressedArray, len);
        } catch (Exception e) {
            len = len * 2;
            i--;
            if (LOGGER.isInfoEnabled()) {
                LOGGER.info("decompress Error: extInfo ={} ", extInfo, e);
            }

        }

    }

    throw new ItemException(1, "decompress error");
}

/**
 * 解压一个数组
 *
 * @param finalCompressedArray 压缩后的数据
 * @param length               原始数据长度, 精确的长度,不能大,也不能小。
 * @return
 */
private static byte[] decompress(byte[] finalCompressedArray, int length) {
    byte[] desc = new byte[length ];
    int decompressLen = decompressor.decompress(finalCompressedArray, desc);

    byte[] result = new byte[decompressLen];
    System.arraycopy(desc,0,result,0,decompressLen);
    return result;
}