LZ4 and Zstd for Java

Question

Is there any Best Java Compression Library available for LZ4 and ZStd. I have tried with apache commons ( which is zstd-jni implementation)

            String fileURL = TestFileUtil.getFileURL(TestFileCategory.SMALL);
            String outputFileName = TestFileUtil.BASE_DIR+"/zstd-"+(Math.random()*10)+".x";
            System.out.println(Paths.get(fileURL));
            printFileInfo(fileURL);
            StopWatch watch = new StopWatch();          
            InputStream in = Files.newInputStream(Paths.get(fileURL));
            OutputStream fout = Files.newOutputStream(Paths.get(outputFileName));
            BufferedOutputStream out = new BufferedOutputStream(fout);
            ZstdCompressorOutputStream zOut = new ZstdCompressorOutputStream(out);
            int buffersize = 1024*4;
            watch.mark();
            final byte[] buffer = new byte[buffersize];
            int n = 0;
            while (-1 != (n = in.read(buffer))) {
                zOut.write(buffer, 0, n);
            }
            zOut.close();
            in.close();

But this code doesnt work it throws

Exception in thread "main" java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStream
    at org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream.<init>(ZstdCompressorOutputStream.java:83)
    at com.zoho.test.testzstd.main(testzstd.java:28)
Caused by: java.lang.ClassNotFoundException: com.github.luben.zstd.ZstdOutputStream
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 2 more

and for LZ4 I just replace ZStdCompressorOutputStream with

FramedLZ4CompressorOutputStream lzOut = new FramedLZ4CompressorOutputStream(out);

But it tooks almost 2hrs (not yet completed ) to compress 2.4GB (csv) file. Is there anything wrong with the code? or anyother suggestions?

Cyan · Answer 1 · 2020-06-29T21:58:18.263

4

For zstd, Java has @luben's zstd-jni : https://github.com/luben/zstd-jni

If you prefer Java native, @martint has a complete port in Java (though with less features) at : https://github.com/airlift/aircompressor/tree/master/src/main/java/io/airlift/compress/zstd

For LZ4, a common binding is : https://github.com/lz4/lz4-java , though if you prefer an implementation which is compatible with the lz4 CLI, the Apache Commons version is preferable.

These libraries generally cannot be swapped by just changing one invocation name. Refer to their documentations or examples in order to understand how they work.

edited Jun 29 '20 at 21:58

answered Jun 29 '20 at 21:32

Cyan

13,248
8
43
78

1

Do you have any specific example of String compression using luben library? I am really finding it hard to get one. Tried this but it doesn't work: byte[] zstdBytes = Zstd.compress(payload.getBytes(StandardCharsets.UTF_8)); ZstdInputStream zstdInputStream= new ZstdInputStream(new ByteArrayInputStream(zstdBytes)); – Raj Feb 04 '21 at 02:56
Yep, that link is answered by me to my own question :) I was able to get over my problem. Thank you for taking time to respond to my question :thumbup – Raj Feb 05 '21 at 23:46
can i pls provide some code or post answer to this question, so we can understand in better way – Gaurav Mandlik Feb 14 '22 at 12:48
how can I decompress this .7z file in custom file location – Gaurav Mandlik Feb 14 '22 at 12:49

score 0 · Answer 2 · answered Mar 10 '23 at 06:24

import com.github.luben.zstd.Zstd;

public class ZSTD {

    public static void main(String[] args) {

        // Zstd zstd = new Zstd();

        System.out.println("Default compress level：" + Zstd.defaultCompressionLevel());

        try {
            //1.File operation
            byte[] src = File2ByteArray.toByteArray("D:\\works\\ZSTD\\data\\data.bin1");
            src = File2ByteArray.toByteArray("D:\\works\\ZSTD\\data\\aa.txt.zst");
            
            byte[] des = new byte[src.length/4];
            
            System.out.println("src length" + src.length);
            
            Zstd.decompress(des, src);
            
            System.out.println( "result：" +new String(des) );
            
            //2

            String data = "ABC";
            byte[] strSrc = Zstd.compress(data.getBytes());

            //byte[] strDest = new byte[ strSrc.length ];
            byte[] strDest = new byte[strSrc.length/4];
            Zstd.decompress( strDest, strSrc );
            
        } catch (Exception ex) {
            ex.printStackTrace();
        }

    }
}

LZ4 and Zstd for Java

2 Answers2

Linked