Stack Overflow
Stack Overflow

Questions tagged [snappy]

Snappy is a compression algorithm for byte streams and a library implementing this algorithm. The standard distribution includes bindings for C and C++; there are third-party bindings for many other languages.

Snappy does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.

Snappy is widely used inside Google, in everything from BigTable and MapReduce to Google's internal systems.

366 questions
0
votes
1 answer

mapreduce using snappy data order error

I meet a problem when I read snappy file on HDFS to mapreduce. I have set job.setInputFormatClass(TextInputFormat.class); in job. Then I get the line value in mapper like this protected void map(Object key, Text value, Context context) …
huyang
  • 1
  • 3
0
votes
0 answers

How to use Snappy Compression And Decompression in hadoop

i am using following code for compression Configuration conf = new Configuration(); conf.setBoolean("mapred.compress.map.output", true); conf.set("mapred.map.output.compression.codec","org.apache.hadoop.io.compress.SnappyCodec"); using…
raju vaishnav
  • 751
  • 1
  • 6
  • 17
0
votes
1 answer

Python cant see installed module

I downloaded Snappy library sources for working with compression and everything was great on one machine, but it didn't work on another machine. They have completely same configurations of hardware/OS + python 2.7.3. All I was doing is "./configure…
dimon222
  • 162
  • 2
  • 20
0
votes
1 answer

Reading Values from Hive using Sqoop and Snappy Compression

I am looking to use an alternate way to compress the files for read/write performance, and one of the avenues I have explored is through the use of Snappy compression. So far, it has been so good, and have been able to get it into HDFS and…
theMJof91
  • 126
  • 2
  • 10
0
votes
1 answer

java.library.path not honored with maven-shade-plugin

If I have code like this: Configuration conf = new Configuration(); LoadIncrementalHFiles lihf; lihf = new LoadIncrementalHFiles(conf); lihf.doBulkLoad(/*proper args*/) This works fine with -Djava.library.path=/usr/lib/hadoop/lib/native/…
shubh
  • 71
  • 4
0
votes
1 answer

can't get hadoop to see snappy

i'm on rhel7 64bit. I managed to apparently build the hadoop 2.4.1 distribution from source. before that, i built snappy from source and installed it. then i build the hadoop dist. with mvn clean install -Pdist,native,src -DskipTests -Dtar…
Majid alDosari
  • 133
  • 1
  • 2
  • 13
0
votes
1 answer

How can I run OrientDB without Snappy?

We are using OrientDB in its embedded Java mode (not as a separate server process), and would like to avoid having Snappy executed from /tmp (for security reasons). My understanding is that Snappy is for compression. I have found a couple…
Kevin Smith
  • 3
  • 1
0
votes
1 answer

How to make my project files with static library (Snappy)

I have project using Snappy library and makefile for it: CXX=g++ CXXFLAGS=-c -Wall LFLAGS= OBJS=main.o Utilities.o FramingFormat.o Crc32.o snappy.out: $(OBJS) $(CXX) $(LFLAGS) $^ -o $@ $(OBJS): %.o:%.cpp $(CXX) $(CXXFLAGS) $< -o…
CppMonster
  • 1,216
  • 4
  • 18
  • 35
0
votes
0 answers

Generate PDF with snappy bundle: works fine in service, doesn't work in containerAwareCommand symfony2

I'm writing an application that collects analytics data, and builds up a HTML page with statistics and Google Charts. When that is done, I use snappyBundle to convert the HTML page (with CSS) to a PDF document. So far, everything has worked fine.…
Tars
  • 1
  • 2
0
votes
1 answer

SnappyFramedInputStream vs SnappyInputStream

I'd like to know implementation/performance differences between Java SnappyFramedInputStream and SnappyInputStream, as well as best usage scenarios for each one of them. (Same for their OutputStream counterparts). Thanks in advance!
fps
  • 33,623
  • 8
  • 55
  • 110
0
votes
1 answer

Modify Grep to parse Sequence/Snappy files

I am trying to get the Grep example bundled with CDH to read Sequence/Snappy files. By default, the program throws errors trying to read Sequence/Snappy files: java.io.EOFException: Unexpected end of block in input stream at…
Joe Nate
  • 159
  • 10
0
votes
0 answers

hbase snappy installation issue

Getting the following issue while setting up Snappy in hadoop/hbase cluster. I've copied both libnappy.so, libhadoop.so to $HBASE_HOME_DIR/lib/native/Linux-amd64-64. Any idea what's going wrong here.. Exception in thread "main"…
Naresh Reddy
  • 350
  • 1
  • 3
  • 7
0
votes
1 answer

Snappy Issue on Mac doing an Avro job on Hadoop

I do work in Mac OSX. I get the following exception trying to run a hadoop job which uses Avro files on a hadoop cluster. I use Java 1.6 and the Snappy version which I use is the 1.0.4.1 which is not the last one but I cant change it because of the…
mettirdas
  • 23
  • 1
  • 1
  • 8
0
votes
1 answer

StackOverFlow exception when using Netty's SnappyFrameDecoder

I'm trying to create a simple program that uses Netty's SnappyFrameEncoder/Decoder. I created a small java application that uses LocalChannels for the server/client. The client encodes a string using Snappy, and the server decodes the string and…
Sprek
  • 3
  • 1
0
votes
1 answer

Snappy Java Exception after Elastic Search Upgrade

Hi guys I've recently upgraded my Elastic Search from 19.4 to 20.6 and one of the new dependencies it required was snappy Java.. I've tried multiple versions of snappy but no matter what it seems like I'm always getting an exception upon startup…
Mantas
  • 3,179
  • 4
  • 20
  • 32
1 2 3
…
24
25