Stack Overflow
Stack Overflow

Questions tagged [lzo]

LZO is a (text) compression algorithm from the Lempel-Ziv family, which favours speed against compression ratio.

LZO is a data compression library which is suitable for data de-/compression in real-time. This means it favours speed over compression ratio.

LZO is written in ANSI C. Both the source code and the compressed data format are designed to be portable across platforms.

LZO implements a number of algorithms with the following features:

Decompression is simple and very fast. Requires no memory for decompression. Compression is pretty fast. Requires 64 kB of memory for compression. Allows you to dial up extra compression at a speed cost in the compressor. The speed of the decompressor is not reduced. Includes compression levels for generating pre-compressed data which achieve a quite competitive compression ratio. There is also a compression level which needs only 8 kB for compression. Algorithm is thread safe. Algorithm is lossless. LZO supports overlapping compression and in-place decompression.

LZO and the LZO algorithms and implementations are distributed under the terms of the GNU General Public License (GPL) .

122 questions
2
votes
1 answer

How to write to hadoop hdfs using lzo compression without map reduce

I am writing to hadoop hdfs. The file has to be compressed using lzo. Also the file will be appended in realtime. The source file is a gzip file that is not present in hadoop. A batch processes this gzip file, and then does lzo compression and…
Krishnan Prakash
  • 21
  • 1
  • 2
2
votes
1 answer

Hadoop handling compression transparently, but not splitting LZO

it looks as if Hadoop handles compression transparently (when was this introduced, I don't remember it on 0.20.203) when using TextInputFormat. Unfortunately, when using LZO compression, Hadoop doesn't use the LZO index file to make the file…
schmmd
  • 18,650
  • 16
  • 58
  • 102
1
vote
2 answers

How can i install python-lzo-1.08?

I need to install python-lzo-1.08. When i try to do it from gz-file, i getting error: NameError: name 'CURL_DIR' is not defined OS: win7 I can't anywhere find windows installer (google taking only gz or broken links to msi/exe). Maybe anybody have…
fenk
  • 13
  • 1
  • 3
1
vote
2 answers

java.io.IOException: No LZO codec found, cannot run

I tried to use lzo in my hive script, but got this error message. It seemed that I did not have the class for lzo in the classpath. Did anyone else meet this problem before, how to fix this problem, maybe what I need to know is that where I can get…
frack
  • 49
  • 1
  • 2
  • 4
1
vote
0 answers

How to read LZO compressed files in Pyspark

I am using PySpark 3.1.2 with virtual environment. I am trying to read files compressed with lzo but i cant find proper documentation on how to do that, i understand that for licensing issue, the lzo codec needs to be added manually to spark. But i…
Abdullah Odibat
  • 21
  • 3
1
vote
2 answers

How to install various compression codecs like LZO and BROTLI on pyspark?

For writing a parquet file and compressing it with LZO codec, I wrote the following code - df.coalesce(1).write.option("compression","lzo").option("header","true").parquet("PARQUET.parquet") But, I am getting this error - Caused by:…
Techie Baba
  • 53
  • 9
1
vote
0 answers

Prepraing lzo or lz4 files for Spark

I'm trying to choose the right format for file exchange with my spark application. I use Spark 2.4.7 + Haddop 2.10 on Kubernetess. My app downloads CSV file from S3 and process it. The file is provided by a 3rd party company. I was thinking about…
Matzz
  • 670
  • 1
  • 7
  • 17
1
vote
0 answers

Decompress LZO indexed files

I need to decompress some lzo indexed files. First, I've tried to decompress the first lzo file with lzop but in the first line of the fie i have some extra byte. How I can decompress the file correctly?
znndrd
  • 79
  • 1
  • 8
1
vote
1 answer

How do I write a LZ compressed string to text file using JXA?

I am trying to write a JXA script in Apple Script Editor, that compresses a string using the LZ algorithm and writes it to a text (JSON) file: var story = "Once upon a time in Silicon Valley..." var storyC = LZString.compress(story) var…
PE B
  • 65
  • 4
1
vote
0 answers

Reading LZO-compressed csv files in tensorflow

I have a collection of csv files compressed in LZO format that I want to import into TensorFlow. However, if I try to read it as I would read uncompressed file, i.e., using def parse_csv(line): columns = tf.decode_csv(line,…
Gianluca Micchi
  • 1,584
  • 15
  • 32
1
vote
1 answer

Decompress MiniLZO String with Python 2.7/python-lzo 1.11

I am writing a Python 2.7 program which interacts with a 3rd party TCP/IP-based API by sending and receiving data packets through a socket connection. The data packets consists of the following parts: A header, which is 10 bytes long, is inserted…
Hiromichi
  • 11
  • 1
1
vote
0 answers

How to decompress lzo compressed byte array in java?

I am new to LZO compression and decompression. I'm trying to use this lzo-java library. Input Information : I have one byte array which is in compressed format. This byte array I want to decompress and finally I want decompressed byte array. NOTE :…
ketan
  • 2,732
  • 11
  • 34
  • 80
1
vote
1 answer

Importing a lzo file into java spark as dataset

I have some data in tsv format compressed using lzo. Now, I would like to use these data in a java spark program. At the moment, I am able to decompress the files and then import them in Java as text files using SparkSession spark =…
Gianluca Micchi
  • 1,584
  • 15
  • 32
1
vote
0 answers

decoding lzo / lz4 broadcast data

Post fetching UDP packets using tcpdmp on the specified port, I'm unable to decode/decompress the data. The source says this data is compressed using lz4/lzo compression algorithm. 15:17:10.915293 IP (tos 0x0, ttl 64, id 3954, offset 0, flags…
Sarthak Bajaj
  • 13
  • 4
1
vote
2 answers

LZO compress char*

I have installed LZO on my Ubuntu machine and I would like to use ti to compress a char* type string. In the example files I have found this code snippet (I have already edited it just a little for my application): int r; lzo_bytep in; …
Richard Knop
  • 81,041
  • 149
  • 392
  • 552
1 2 3
…
8 9