Stack Overflow
Stack Overflow

Questions tagged [lz4]

LZ4 is a lossless data compression algorithm that is focused on compression and decompression speed

LZ4 is a lossless data compression algorithm that is focused on compression and decompression speed. It belongs to the LZ77 family of byte-oriented compression schemes.

Use this tag for questions that have issues with the implementations in one of the native languages or binding like c, c++, java, python.

LZ4 Repository

LZ4 Homepage

Adapted From Wikipedia

166 questions
4
votes
1 answer

Decompressing LZ4 compressed data in Spark

I have LZ4 compressed data in HDFS and I'm trying to decompress it in Apache Spark into a RDD. As far as I can tell, the only method in JavaSparkContext to read data from HDFS is textFile which only reads data as it is in HDFS. I have come across…
shoopdelang
  • 985
  • 2
  • 9
  • 20
3
votes
1 answer

Invalid magic number when decompressing using javascript. Headers not added when compressing using lz4 in golang

I have an gin api in which i compress a string using lz4 frame compression and send the result in the response to a frontend. In the javascript I fetch that api and decompress the data using lz4 package by pierrec, at that time I am getting Invalid…
Anand Samuel
  • 71
  • 2
3
votes
1 answer

Kafka LZ4 decompression causes IllegalAccessError in lz4-java when reading from topic

On the project I am working on currently, we have four spring based apps that are writing and reading from various Kafka topics. Due to the large amount of data being sent, we have decided to enable LZ4 compression. This worked perfectly fine for…
Remus
  • 41
  • 2
3
votes
1 answer

LZ4 not found for CMake on Android Studio

I'm trying to compile OpenVPN library inside Android, but I'm getting this error: C/C++ debug|x86 : CMake Error at /home/user/Android/Sdk/cmake/3.10.2.4988404/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message): Could NOT…
Guerlando OCs
  • 1,886
  • 9
  • 61
  • 150
3
votes
2 answers

How to read a compressed Spark eventLog?

When I try to read a Spark 2.4.4 eventLog compressed with lz4, I obtain an empty DataFrame: cd /opt/spark-2.4.4-bin-hadoop2.7 bin/spark-shell --master=local --conf spark.eventLog.enabled=true --conf spark.eventLog.compress=true --conf…
Mikael Valot
  • 444
  • 4
  • 6
3
votes
1 answer

How to enable LZ4 compression for messagepack content type in Asp.net Core Web API

I'm trying to enable MessagePack content type on a .net Core Web API Project that I'm working on. After some research, Installed this nuget package and added below code in the startup file. Easy enough ! Now I can see msgpack content served through…
Naren
  • 298
  • 2
  • 10
3
votes
0 answers

How to read Spark log files? .Iz4 or .snappy

I want to read some logs but I cannot. So far, I've tried: hadoop fs -text But the only thing I get is: INFO compress.CodecPool: Got brand-new decompressor [.lz4] (same for .snappy) val rawRdd =…
mjbsgll
  • 722
  • 9
  • 24
3
votes
1 answer

Reading an LZ4 compressed text file (mozlz4) in WebExtensions (JavaScript, Firefox)

I'm porting a Firefox Add-on SDK extension to WebExtensions. Previously I could access the browser's search engines, but now I can't, so a helpful user suggested I try reading the search.json.mozlz4 file, which has every installed engine. However,…
CanisLupus
  • 583
  • 5
  • 15
3
votes
0 answers

Windows compilation failure on pip install due to invalid command flags

I was trying to install the lz4 package on my Windows machine from PyPi using pip. This package doesn't have any wheels uploaded so it was going to need to compile the source locally. I have the Visual C++ for Python compiler installed. I…
naltmann
  • 53
  • 4
3
votes
3 answers

Efficient LZ4 multiple file compression using java

I took adrien grand's java repository providing JNI to the original LZ4 native code. I want to compress multiple files under a given input directory, but LZ4 doesn't support multiple file compression like in java zip package so I tried another…
Krishna Sundar
  • 331
  • 3
  • 7
3
votes
1 answer

LZ4 compressed text is larger than uncompressed

I have read that lz4 algorithm is very fast and has pretty good compression. But in my test app compressed text is larger than the source text. What is the problem? srand(time(NULL)); std::string text; for (int i = 0; i < 65535; ++i) …
user2123079
  • 656
  • 8
  • 29
3
votes
1 answer

LZ4 match search algorithm (fast scan)

I implemented a LZ77/LZ4 (no entropy encoding) based compression algorithm based on hash chains of infinite depth. It works well and its speed is acceptable, but its compression ratio is close to LZ4. Reading documentation and browsing source code…
user3368561
  • 779
  • 6
  • 18
3
votes
0 answers

LZ4 module not recognized in python

I successfully installed LZ4 in Ubuntu but still in Python while running this command: import lz4 I am getting error no module found .
shilpi_agrawal
  • 108
  • 1
  • 9
3
votes
2 answers

How to correctly implement LZ4, Snappy or equivalent compression techniques in Java?

I've tried implementing Java version of LZ4 into a search engine kind of program trying to search data from large text files. I simply compressed the outputstream and stored it into txt files or files without names. However, I realized the…
kdenz
  • 671
  • 1
  • 7
  • 16
2
votes
0 answers

reading data compressed by NSMutableData using the lz4 library in python

I'm compressing random data using NSMutableData (in Swift) as follows: import Foundation let n = 1024 let numbers = (0..
Taylor
  • 5,871
  • 2
  • 30
  • 64
1 2
3
…
10 11