1

I have written a native hadoop compression codec. To let it work with hadoop I need to compile native (C code) it to libhadoop.so.

How can I achieve this?

samarth
  • 3,866
  • 7
  • 45
  • 60

1 Answers1

0

You don't need to compile this into libhadoop.so:

  • Compile your own .so and distribute to your cluster nodes (into the same directory as the current libhadoop.so
  • I assume you've also written your own CompressionCodec (similar to GzipCodec) - add a static block to this code which tries to load the library using System.loadLibrary("mylibrary"); (for a lib named libmylibrary.so).
  • Amend your cluster configuration to include your new compression codec class in the registered list of codecs (amend the io.compression.codecs configuration property):
  • Restart your task trackers

As a reference, you can follow the implementation and configuration notes for the Google Snappy codec:

Chris White
  • 29,949
  • 4
  • 71
  • 93
  • Thanks, that is really a good option and I have decided to go with it.But I came across other isuue,my codec loads liblzma.so file.But when I install this lzma library with apt-get install ..the liblzma.so of size 0 is created and my codec is not able to load it... – samarth Apr 25 '12 at 06:36
  • Have you tried removing the re-installing, or install from source? – Chris White Apr 25 '12 at 10:15
  • Have tried reinstalling but no luck...not aware of install from source..please let me know how to do it? – samarth Apr 25 '12 at 13:51