Stack Overflow
Stack Overflow

Questions tagged [lzo]

LZO is a (text) compression algorithm from the Lempel-Ziv family, which favours speed against compression ratio.

LZO is a data compression library which is suitable for data de-/compression in real-time. This means it favours speed over compression ratio.

LZO is written in ANSI C. Both the source code and the compressed data format are designed to be portable across platforms.

LZO implements a number of algorithms with the following features:

Decompression is simple and very fast. Requires no memory for decompression. Compression is pretty fast. Requires 64 kB of memory for compression. Allows you to dial up extra compression at a speed cost in the compressor. The speed of the decompressor is not reduced. Includes compression levels for generating pre-compressed data which achieve a quite competitive compression ratio. There is also a compression level which needs only 8 kB for compression. Algorithm is thread safe. Algorithm is lossless. LZO supports overlapping compression and in-place decompression.

LZO and the LZO algorithms and implementations are distributed under the terms of the GNU General Public License (GPL) .

122 questions
4
votes
2 answers

describe extended table in Hive

I am storing the Table as a SequenceFile format and I am setting the below commands to enable Sequence with BLOCK Compression- set mapred.output.compress=true; set mapred.output.compression.type=BLOCK; set…
arsenal
  • 23,366
  • 85
  • 225
  • 331
3
votes
1 answer

Reasonable compression block-size

What is a reasonable minimum compression block-size for compression ratio when using LZO-like algorithm? I expect that compressing 32B would be useless but compressing 512B might be good. Am I too far? Please, no "check yourself answers" :)
Cartesius00
  • 23,584
  • 43
  • 124
  • 195
3
votes
1 answer

compile 64-bit version of lzo.dll

[Update] I've since compiled successfully and anyone else chasing these binaries can download from here I'm compiling version 2.06 of lzo by issuing the following command from the Visual Studio Command Prompt (2010) b\win64\vc_dll.bat which…
wal
  • 17,409
  • 8
  • 74
  • 109
3
votes
2 answers

LZO Decompression Buffer Size

I am using MiniLZO on a project for some really simple compression tasks. I am compressing with one program, and decompressing with another. I'd like to know how much space to allocate for the decompression buffer. I am fine with over-allocating…
Benson
  • 22,457
  • 2
  • 40
  • 49
3
votes
2 answers

Newest version of lzo.dll for use with lzo.net (in memory compression)

for use with lzo-net ( http://lzo-net.sourceforge.net/ ) I am looking for a new version of the lzo.dll file. Mine is from 2004. The newest file should be 2.0.6 ( http://www.oberhumer.com/opensource/lzo/download/ ) But I only have visual studio and…
Chris
  • 4,325
  • 11
  • 51
  • 70
3
votes
2 answers

Compression Algorithm for Small Amounts of Data

I have a server-side program that generates JSON for a client. A few of my colleagues have suggested using zip/gzip compression in order to reduce the amount of data that sending over the wire. However, when tested against one of my average JSON…
Patrick Perini
  • 22,555
  • 12
  • 59
  • 88
3
votes
3 answers

Failed to load native-lzo library when setting up Lzo on Cloudera Hadoop

I just followed the steps in the Cloudera Document, and had GPL Extras Parcel installed on the cluster as well as configured HDFS service via Cloudera Manager. But an error ocurred when trying to read .lzo files on HDFS: $hadoop fs -text…
Paul Lam
  • 131
  • 1
  • 11
3
votes
1 answer

Ignore errors in linux pipelines

I have a long list of filenames in filenames.txt file. These files are lzo compressed and I use lzop to decompress them for further processing in a pipeline. cat filenames.txt | (xargs lzop -dc || true) | python lineprocessor.py > output.txt So…
Ash
  • 3,428
  • 1
  • 34
  • 44
3
votes
0 answers

How do I increase HTTPfs's timeout parameter

Right now, my ingest process using HTTPfs is timing out around a minute. Can this timeout setting be adjusted? 9:52:17.486 PM ERROR org.apache.hadoop.security.UserGroupInformation PriviledgedActionException as:manager (auth:PROXY) via httpfs…
Carl Sagan
  • 982
  • 1
  • 13
  • 34
3
votes
2 answers

Are there benchmarks of Java LZO vs. Deflate vs. Gzip available?

The (very outdated) page for LZO contains a link to a Java implementation. There is also the vague promise of "New ... Java ... interfaces ... sometime in the near future.", but I believe that has been there for a few years. This makes me think…
jsight
  • 27,819
  • 25
  • 107
  • 140
3
votes
1 answer

Hadoop: How to output different format types in the same job?

I want to output gzip and lzo formats at the same time in one job. I used MultipleOutputs, and add two named outputs like this: MultipleOutputs.addNamedOutput(job, "LzoOutput", GBKTextOutputFormat.class, Text.class,…
thomaslee
  • 407
  • 1
  • 7
  • 21
3
votes
2 answers

Error using elephant-bird input format with Hadoop Streaming

I'm trying to use an input format from Elephant Bird in my Hadoop Streaming script. In particular, I want to use the LzoInputFormat and eventually the LzoJsonInputFormat (working with Twitter data here). But when I try to do this, I keep getting an…
raynach
  • 577
  • 1
  • 6
  • 10
2
votes
1 answer

LZO compression for Hbase

I'm trying to setup LZO compression for Hbase. But I'm getting a problem while creating build.xml. Logs are following below: anonymouse@hbase:~/omalley-hadoop-gpl-compression-d9deaa2$ sudo ant compile-nativeBuildfile: build.xml ivy-download: …
Ali Raza
  • 1,215
  • 3
  • 15
  • 26
2
votes
1 answer

using lzo1c_1_compress method in lzo dll

Hi I want to compress a file using lzo in C# I want to compress it with lzo1c_1_compress but I don't know why its not working at all and make the program crash, here is my code using System; using System.Diagnostics; using…
sanab3343
  • 154
  • 1
  • 11
2
votes
1 answer

CompressionTest for LZO fails

I sincerely thank you for reading my post. I'm trying to install LZO 2.03 compression codec on HBase on my server (running Xeon CPUs). I'm currently running Hadoop 0.20.1 with HBase 0.90.2. I've followed the guidelines from…
Chul Kwon
  • 167
  • 1
  • 3
  • 11
1
2
3
…
8 9