Stack Overflow
Stack Overflow

Questions tagged [hadoop-lzo]

Hadoop-LZO is a project to bring splittable LZO compression to Hadoop.

Source

Hadoop-LZO is a project to bring splittable LZO compression to Hadoop. LZO is an ideal compression format for Hadoop due to its combination of speed and compression size. However, LZO files are not natively splittable, meaning the parallelism that is the core of Hadoop is gone. This project re-enables that parallelism with LZO compressed files, and also comes with standard utilities (input/output streams, etc) for working with LZO files.

21 questions
0
votes
0 answers

hive sql add sort or distribute then the result file size bigger than before

My hive tables are all lzo compressed type. I have two hive-sql like this: [1] set hive.exec.compress.output=true; set mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec; insert overwrite table a partition(dt='20160420') select…
user1654333
  • 3
  • 2
0
votes
1 answer

LZO files issue on S3

I have 3 LZO compressed files and their corresponding index files in HDFS. Permission Owner Group Size Replication Block Size Name -rw-r--r-- alum supergroup 0 B 3 128 MB _SUCCESS -rw-r--r-- alum supergroup 192.29 MB 3 128…
user1570824
  • 37
  • 1
  • 6
0
votes
1 answer

Created Hive table with lzo compression, cant locate file with extension .lzo

I created a Hive table by setting the following Properties on hive command prompt: SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec SET hive.exec.compress.output=true SET…
user2176576
  • 734
  • 3
  • 14
  • 39
0
votes
0 answers

java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path

I got: java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path so i followed: [http://wiki.apache.org/hadoop/UsingLzoCompression] so after running $ant compile-native I get many errors: $ ant compile-native Buildfile:…
Jas
  • 14,493
  • 27
  • 97
  • 148
0
votes
1 answer

Do we need to create an index file (with lzop) if compression type is RECORD instead of block?

As I understand, an index file is needed to make the output Splitable. If mapred.output.compression.type=SequenceFile.CompressionType.RECORD, do we still need to create an Index file?
Apurv
  • 4,458
  • 2
  • 21
  • 31
0
votes
2 answers

NoSuchElementException while exporting data in lzo compression from hdfs to mysql using sqoop

I am trying to export some data from hdfs to mysql using sqoop. The problem is when I try to export a uncompressed file it exports properly but if I try to export the same file compressed with lzo compression, the sqoop job fails. I am trying it in…
Dipayan
  • 113
  • 1
  • 7
1
2