Stack Overflow
Stack Overflow

Questions tagged [snappy]

Snappy is a compression algorithm for byte streams and a library implementing this algorithm. The standard distribution includes bindings for C and C++; there are third-party bindings for many other languages.

Snappy does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.

Snappy is widely used inside Google, in everything from BigTable and MapReduce to Google's internal systems.

366 questions
3
votes
1 answer

Configuring the mongodb node.js driver to use snappy compression

We've recently upgraded our MongoDB replica set to v3.4.4; I notice that this release now supports compressed network communication with snappy. I've got the set members set to enable this, so that they communicate with each other this way, and the…
Thomas J. Girsch
  • 41
  • 2
3
votes
2 answers

Unable to decompress snappy file, even after specifying -Djava.library.path=/*/*/

Exception in thread "main" java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) at…
ShriRam
  • 31
  • 1
3
votes
1 answer

Hive results save as parquet file

I'm trying to create a snappy.parquet file from a Hive table. Its a big partitioned table just need small part of it. Doing this: set parquet.compression=SNAPPY; set hive.exec.compress.output=true; set hive.exec.compress.intermediate=true; set…
lacerated
  • 375
  • 1
  • 4
  • 17
3
votes
3 answers

How to put file to HDFS with Snappy compression

I am working for a client where I should put some files to HDFS with Snappy compression. My problem is snappy codec is not defined in mapred-site.xml or hdfs-site.xml Somehow I have to put files preferably using hdfs put command and they should be…
Canburak Tümer
  • 993
  • 17
  • 36
3
votes
1 answer

Native snappy library not available: this version of libhadoop was built without snappy support

I got the above error when I was using MLUtils saveAsLibSVMFile. tried various approaches like below but nothing worked. /* conf.set("spark.io.compression.codec","org.apache.spark.io.LZFCompressionCodec") */ /* …
Run2
  • 1,839
  • 22
  • 32
3
votes
0 answers

Spark Streaming Error with snappy compression codec

I am getting the following exception on running my spark streaming job. The same job has been running fine since long and when I added two new machines to my cluster I see the job failing with the following exception. 16/02/22 19:23:01 ERROR…
Abhishek Anand
  • 181
  • 1
  • 8
3
votes
3 answers

how does hive create table using parquet and snappy

I know the syntax for creating a table using parquet but I want to know what does this mean to create tables using parquet format and compressed by snappy and how does we do that ? please help me with an example syntax. What is the advantage by…
samuob
  • 31
  • 1
  • 1
  • 2
3
votes
1 answer

Fastest Decompression Algorithm?

I heard Google snappy is a fast de/compression library. What does Snappy library use as their algorithm? Is there some algorithm that is faster in decompressing? The compression rate and encoding is not my interest, but real time decompression…
John Doyle
  • 898
  • 2
  • 9
  • 22
3
votes
1 answer

PDF Generation with Snappy- Laravel

So, snappy does an amazing job for creating PDFs for my Laravel site. The problem is the login. Snappy can PDF all the pages before the login but if I try to pdf the url of a page that appears only after login, then it PDFs the login page. It must…
Panos
  • 61
  • 1
  • 5
3
votes
0 answers

Spark not working with error "java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path"

I am using the CDH4 build of Spark 1.2.0 to submit a jar of Scala code built using sbt assembly. Both spark shell and spark submit are failing with below error. java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path at…
Varun
  • 93
  • 1
  • 8
3
votes
0 answers

java fatal error when using hadoop snappy

I just installed hadoop snappy and when I tried to use hadoop fs -text /xxxx/xxx/xx/xx.snappy command to see the content of a snappy file in hdfs, it throws a fatal error: 2015-01-06 13:06:39.248 java[22668:285995] Unable to load realm info from…
Woaibanzhuan
  • 127
  • 7
3
votes
0 answers

Decompressing Snappy Files Missing Stream Identifier Chunk and CRC-32C Checksum

The iWork 2013 file format includes .iwa (iWork Archive) files stored within a .zip file. "IWA files are stored in Snappy's framing format, though they do not adhere rigorously to the spec. In particular, they do not include the required Stream…
Jacob Quisenberry
  • 1,131
  • 3
  • 20
  • 48
3
votes
3 answers

kafka node js client compression issue with snappy

I am using kafka-node (https://github.com/SOHU-Co/kafka-node) consumer to retrieve data. I think the data which I get is compressed with SNAPPY. How do I decompress the data after I get it. I tried using node-snappy…
CKR
  • 213
  • 4
  • 8
3
votes
2 answers

How to correctly implement LZ4, Snappy or equivalent compression techniques in Java?

I've tried implementing Java version of LZ4 into a search engine kind of program trying to search data from large text files. I simply compressed the outputstream and stored it into txt files or files without names. However, I realized the…
kdenz
  • 671
  • 1
  • 7
  • 16
3
votes
1 answer

Snappy GLIBCXX_3.4.9 error when using datastax 4 cassandra

Complete error while starting cassandra. java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at…
Amey Jah
  • 913
  • 3
  • 11
  • 24
1 2 3
…
24 25