1

Trying to read snappy compressed table using Presto on CDH4 cluster. Hive can read the table, but Presto complains about missing native library.

Adding java.library.path from CDH doesn't seem to help. (added to jvm.config)

-Djava.library.path=/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/lib/native

Here is the stack trace

Caused by: java.lang.RuntimeException: native snappy library not available: SnappyCompressor has not been loaded.
at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:68) ~[na:na]
at org.apache.hadoop.io.compress.SnappyCodec.createDecompressor(SnappyCodec.java:196) ~[na:na]
at org.apache.hadoop.hive.ql.io.CodecPool.getDecompressor(CodecPool.java:35) ~[na:na]
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1314) ~[na:na]
at org.apache.hadoop.hive.ql.io.RCFile$Reader.<init>(RCFile.java:1154) ~[na:na]
at org.apache.hadoop.hive.ql.io.RCFile$Reader.<init>(RCFile.java:1133) ~[na:na]
at org.apache.hadoop.hive.ql.io.RCFileRecordReader.<init>(RCFileRecordReader.java:102) ~[na:na]
at org.apache.hadoop.hive.ql.io.RCFileInputFormat.getRecordReader(RCFileInputFormat.java:57) ~[na:na]
at com.facebook.presto.hive.HiveRecordSet$1.call(HiveRecordSet.java:185) ~[na:na]
  ...

Library dir

ls -l /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/lib/native
total 564
-rwxr-xr-x 1 root root 133184 Sep  6 00:12 libhadoop.a
-rwxr-xr-x 1 root root 193020 Sep  6 00:12 libhadooppipes.a
lrwxrwxrwx 1 root root     18 Sep  6 08:26 libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x 1 root root  81680 Sep  6 00:12 libhadoop.so.1.0.0
-rwxr-xr-x 1 root root  58600 Sep  6 00:12 libhadooputils.a
-rwxr-xr-x 1 root root  77476 Sep  6 00:12 libhdfs.a
lrwxrwxrwx 1 root root     18 Sep  6 08:26 libsnappy.so -> libsnappy.so.1.1.3
lrwxrwxrwx 1 root root     18 Sep  6 08:26 libsnappy.so.1 -> libsnappy.so.1.1.3
-rwxr-xr-x 1 root root  23904 Sep  6 00:12 libsnappy.so.1.1.3
Igor Semenko
  • 459
  • 1
  • 7
  • 18

1 Answers1

2

Presto 0.54 supports Snappy compression.

David Phillips
  • 10,723
  • 6
  • 41
  • 54
  • Thanks, looking forward to it. Are you using any compression at Facebook? – Igor Semenko Nov 16 '13 at 06:10
  • With over 300 PB in our data warehouse you can be certain we are using compression :) The majority of our data is currently stored using RCFile with deflate (zlib) and we are moving to ORCFile which also uses zlib. – David Phillips Nov 18 '13 at 05:52
  • I updated the answer to reflect that Snappy compression is now supported as of Presto 0.54. – David Phillips Nov 28 '13 at 00:42