Stack Overflow
Stack Overflow

Questions tagged [lz4]

LZ4 is a lossless data compression algorithm that is focused on compression and decompression speed

LZ4 is a lossless data compression algorithm that is focused on compression and decompression speed. It belongs to the LZ77 family of byte-oriented compression schemes.

Use this tag for questions that have issues with the implementations in one of the native languages or binding like c, c++, java, python.

LZ4 Repository

LZ4 Homepage

Adapted From Wikipedia

166 questions
2
votes
3 answers

Dictionary-based compression in Python to compress short strings individually

I have millions of < 20 char strings, and I want to compress each of them individually. Using zlib or lz4 on each string individually doesn't work: the output is bigger than the input: inputs = [b"hello world", b"foo bar", b"HELLO foo bar world",…
Basj
  • 41,386
  • 99
  • 383
  • 673
2
votes
1 answer

Why doesn't lz4 keep small/uncompressible values uncompressed?

When compressing small values (< 500 bytes or so), and for uncompressible random values, lz4 returns data that is much larger than the original value (e.g. 27 bytes from 4). When a large amount of such values is compressed separately (e.g. in a…
HoverHell
  • 4,739
  • 3
  • 21
  • 23
2
votes
0 answers

LZ4 Compression difference linux wml vs windows

I tried to compress a gif image with LZ4. The results seem to differ in the header for Linux wml and Windows in the first 77 bytes, the rest of the file is identical. I ran the same code in both cases. Linux WML also has 2 bytes more than windows.…
Meng
  • 191
  • 9
2
votes
0 answers

Decompressing load on node js using LZ4

Our node server serves more than 30,000 users simultaneously. The load size is pretty small, it varies 1-4K, the average is 2K of raw data. At the moment we are using deflate compressing algorithm and using Zlib lib to decompress it. The code is…
Denis Voloshin
  • 768
  • 10
  • 32
2
votes
2 answers

How can I decompress a jsonlz4 Mozilla Firefox session file?

I'm trying to decode the jsonlz4 file that contains my saved tabs from Firefox (saved in %appdata%\Mozilla\Firefox\Profiles\xxxxxx.default-release\sessionstore-backups\recovery.jsonlz4, in the hope of eventually being able to parse the json and…
localhost
  • 1,253
  • 4
  • 18
  • 29
2
votes
1 answer

Data compression using Arrow.jl in Julia

I tried to compress data using Arrow.jl. However, the test run using the below code didn’t show any size reduction (or compression). May I seek advice on my implementation, like is there something I am doing wrong? Code: using CSV, DataFrames,…
Mohammad Saad
  • 1,935
  • 10
  • 28
2
votes
1 answer

Streaming contents from lz4 file using InputStreamReader - Stream Corrupted - Java

I am trying to stream data from the lz4 compressed file and write it to StringBuilder. But I am getting Stream is corrupted exception. Below is my code. import java.io.ByteArrayInputStream; import java.io.File; import…
user2731629
  • 402
  • 1
  • 7
  • 17
2
votes
1 answer

.net 5 - Signalr - Messagepack - lz4 - Error: unable to find ext type 98

Got this error un .net core 5 solution with angular client after enabling lz4BlockArray compression. Without compression everithing seems to work pretty fine. Looking around for motivation but i found nothing. Seems like some decoders aren't loaded…
montipirlo
  • 23
  • 2
2
votes
0 answers

LZ4 not compressing data (JavaBindings - org.lwjgl.lz4)

I am using a Java-Binding for the popular LZ4 compression library, in my case org.lwjgl.lz4 with the intend of compressing and managing a variety assets that way. I have working compression and de-compression of my custom format, verified by tests,…
DragonReborn
  • 73
  • 4
2
votes
1 answer

LZ4_decompress_safe vs LZ4_decompress_safe_partial

What is the difference between LZ4_decompress_safe and LZ4_decompress_safe_partial? I understand that LZ4_decompress_safe_partial does partial decompression, but it looks like LZ4_decompress_safe also can do partial decompression or I am wrong. Both…
flashburn
  • 4,180
  • 7
  • 54
  • 109
2
votes
2 answers

LZ4 and Zstd for Java

Is there any Best Java Compression Library available for LZ4 and ZStd. I have tried with apache commons ( which is zstd-jni implementation) String fileURL = TestFileUtil.getFileURL(TestFileCategory.SMALL); String…
Hulk_avenger
  • 21
  • 1
  • 2
2
votes
1 answer

Lz4 compression is not splittable

I am using lz4 compression and write data to a hive table, this table has 20 files and each is 15G on HDFS, and every file name of this table are ending with lz4, eg, part-m-00000.lz4. When I run select count(1) from this table, it kicks off only…
Tom
  • 5,848
  • 12
  • 44
  • 104
2
votes
2 answers

CMakeLists.txt not in root directory, can't do an automatic build

I have an issue with a repository not having its CMakeLists in the root directory, namely https://github.com/lz4/lz4 The CMakeLists.txt is in the subfolder contrib/cmake_unofficial. I already checked similar questions on SO (Is it possible to have…
Tasche
  • 65
  • 6
2
votes
1 answer

How do I should choose my log compression type in apache kafka?

I want to do compress data log in Apache Kafka. How do I know which one to choose? For me, performance and space is important. Server.properties file Compression.type = snappy , gzip , lz4 vb. use.
omerstack
  • 535
  • 9
  • 23
2
votes
0 answers

editing/decompiling/decompressing .lua files

I've been trying to edit the lua files for the game victor vran (made by developers of the tropico series) but when i tried to decompile the files i'd get back errors. So i opened up some of the files in both notepad++ and luaedit, and noticed the…
kaylin1997
  • 25
  • 1
  • 6
1 2 3
…
10 11