Stack Overflow
Stack Overflow

Questions tagged [lzo]

LZO is a (text) compression algorithm from the Lempel-Ziv family, which favours speed against compression ratio.

LZO is a data compression library which is suitable for data de-/compression in real-time. This means it favours speed over compression ratio.

LZO is written in ANSI C. Both the source code and the compressed data format are designed to be portable across platforms.

LZO implements a number of algorithms with the following features:

Decompression is simple and very fast. Requires no memory for decompression. Compression is pretty fast. Requires 64 kB of memory for compression. Allows you to dial up extra compression at a speed cost in the compressor. The speed of the decompressor is not reduced. Includes compression levels for generating pre-compressed data which achieve a quite competitive compression ratio. There is also a compression level which needs only 8 kB for compression. Algorithm is thread safe. Algorithm is lossless. LZO supports overlapping compression and in-place decompression.

LZO and the LZO algorithms and implementations are distributed under the terms of the GNU General Public License (GPL) .

122 questions
2
votes
3 answers

Unable to install OpenVPN on macOS - configure: error: lzo enabled but missing

I'm trying to install OpenVPN on macOS High Sierra I have cloned the github repo: git clone https://github.com/OpenVPN/openvpn And switched to the latest stable branch: git checkout origin release/2.4 But when I tried to build the project…
freedev
  • 25,946
  • 8
  • 108
  • 125
2
votes
1 answer

Can't import lzo files in pyspark

I have a csv file compressed in lzo format and I want to import it into a pyspark dataframe. Were the file not compressed, I would simply do: import pyspark as ps spark = ps.sql.SparkSession.builder.master("local[2]").getOrCreate() data =…
Gianluca Micchi
  • 1,584
  • 15
  • 32
2
votes
2 answers

How to decompress lzo byte array using java-lzo library?

I'm trying to decompress compressed byte array using java-lzo library. I'm following this reference. I added below maven dependency to pom.xml - org.anarres.lzo lzo-core …
ketan
  • 2,732
  • 11
  • 34
  • 80
2
votes
3 answers

How to get LZO to work with a file stream?

I am trying to compress a file stream with LZO and not getting very far. Specifically, I get a segmentation fault when extracting the archive file created by my compressFileWithLzo1x function. My main function and prototype declarations…
Alex Reynolds
  • 95,983
  • 54
  • 240
  • 345
2
votes
3 answers

File not found error when pip installing python-lzo

I'm trying to install python-lzo on mac OS Sierra. I have used brew to install the c lzo library: brew update brew install lzo I then try and install python-lzo using a virtual environment and pip: $ virtualenv ./env $ . ./env/bin/activate $ pip…
emson
  • 10,005
  • 4
  • 24
  • 27
2
votes
1 answer

Hadoop SequenceFile vs splittable LZO

We're choosing the file format to store our raw logs, major requirements are compressed and splittable. Block-compressed (whichever codec) SequenceFiles and Hadoop-LZO look the most suitable so far. Which one would be more efficient to be processed…
k0_
  • 101
  • 3
2
votes
1 answer

Cannot load lzop-compressed files from S3 into Redshift

I am attempting to copy an lzop-compresed file from S3 to Redshift. The file was originally generated by using S3DistCp with the --outputCodec lzo option. The S3 file seems to be compressed correctly, since I can successfully download and inflate it…
fblundun
  • 987
  • 7
  • 19
2
votes
2 answers

Java LZO compression library

I'm trying to use LZO compression library inside my Java program (http://www.oberhumer.com/opensource/lzo/). I could not find a single example how to use it for compression and decompression of data. Can anybody help me with it? Apparently the…
Tina J
  • 4,983
  • 13
  • 59
  • 125
2
votes
1 answer

Why does my LZO indexing take so long on Amazon's EMR when reading from S3?

I have a 30gb lzo file on S3, and I'm using hadoop-lzo to index it with Amazon EMR (AMI v2.4.2), using region us-east1. elastic-mapreduce --create --enable-debugging \ --ami-version "latest" \ --log-uri s3n://mybucket/mylogs \ --name…
Dolan Antenucci
  • 15,432
  • 17
  • 74
  • 100
2
votes
2 answers

Read sequential file - Compressed file vs Uncompressed

I am looking for the fastest way to read a sequential file from disk. I read in some posts that if I compressed the file using, for example, lz4, I could achieve better performance than read the flat file, because I will minimize the i/o…
p.magalhaes
  • 7,595
  • 10
  • 53
  • 108
2
votes
1 answer

processing LZO sequence files with mrjob

I'm writing a task with mrjob to compute various statistics using the Google Ngrams data: https://aws.amazon.com/datasets/8172056142375670 I developed & tested my script locally using an uncompressed subset of the data in tab-delimited text. Once I…
burr
  • 529
  • 5
  • 8
2
votes
2 answers

Hadoop compression : "Loaded native gpl library" but "Failed to load/initialize native-lzo library"

after several try installing Lzo compression for hadoop, I need help because I have really no idea why it doesn't work. I'using hadoop 1.0.4 on CentOs 6. I tried http://opentsdb.net/setup-hbase.html, https://github.com/kevinweil/hadoop-lzo and some…
Brugere
  • 436
  • 4
  • 16
2
votes
1 answer

Hadoop LZO & SnappyCodec error in Hadoop and Hive

I am using Ubuntu-12.04,Hadoop-1.0.2,Hive-0.10.0 while reading data about 1 million records from hive I got error below for query select * from raw_pos limit 10000; WARN util.NativeCodeLoader: Unable to load native-hadoop library for your…
Sagar Nikam
  • 1,708
  • 3
  • 22
  • 35
2
votes
2 answers

hadoop: automatic splittable output from lzo compression

I am setting up lzo codec to use as the compression tool in my hadoop jobs. I know that lzo has the desirable feature of creating splittable files. But I have not found a way to get lzo create splittable files automatically. The blogs I have read so…
nom-mon-ir
  • 3,748
  • 8
  • 26
  • 42
2
votes
1 answer

importing compressed (lzo) data from s3 to hive

I export my DynamoDB tables to s3 as a means of backup (via EMR). When I export, I store the data as lzo compressed file. My hive query is below, but essentially I followed the "To export an Amazon DynamoDB table to an Amazon S3 bucket using data…
rynop
  • 50,086
  • 26
  • 101
  • 112
1 2
3
…
8 9