3

I am using kafka-node (https://github.com/SOHU-Co/kafka-node) consumer to retrieve data. I think the data which I get is compressed with SNAPPY. How do I decompress the data after I get it. I tried using node-snappy (https://github.com/kesla/node-snappy) to decompress the data, but it didn't work.

Is there any option in the library to set the compression to none?

Anyone used kafka-node library to get data from kafka..??

Thanks, chandu

CKR
  • 213
  • 4
  • 8

3 Answers3

2

I also encountered these exact problems. I found a solution, at last! You can use kafkacat ('like netcat for kafka') download here, which requires librdkafka. This enables you to interact with Kafka from the command line using the librdkafka C/C++ library. (no JVM required =D )

Now that we have those dependencies taken care of we can use this fun node.js repo: node-kafkacat

You'll find that there's likely enough documentation between those three libraries to get you started and unlike some of the other kafka-node modules on github, they seem to have been updated fairly recently.

I have only successfully installed on Linux and Mac so far but things are working great with our Apache/Java environment. I'm not the author of any of these packages, btw - just some guy who kept hoping your question would be answered over the last couple weeks.

Jon
  • 157
  • 1
  • 3
  • 11
1

By default, compression is determined by the producer through the configuration property 'compression.type'. Currently gzip, snappy and lz4 are supported. kafka-node should automatically uncompress both gzip and snappy. I just tried it with snappy and this works out of the box. lz4 appears not to be implemented by kafka-node at this point.

Compression can be configured using the configuration property 'compression.type':

If you can't influence the producer, you could therefore consider overriding the topic configuration.

Also, note that kafka-node may not directly return the data as you expect. For instance, my (key, value) pair is of type (integer, integer), and I have to call both message.key.readIntBE() and message.value.readIntBE() to extract the integer values.

Hope this helps!

thedutchy
  • 131
  • 7
0

I have similar problem with LZ4 compression, seems both kafkajs and kafka-node only support gzip, and for snappy and LZ4 you have to install another package. For LZ4 package, kafkajs-lz4 is not working for me, so I have to implement my custom decoder. Simply download lz4 instead of kafkajs-lz4 works for me.

So finally I decided to use kafkajs instead of kafka-node.

const LZ4 = require("lz4");
const LZ4Codec = {
    async compress(encoder) {
        // TODO:
        return LZ4.encode(encoder);
    },

    async decompress(buffer) {
        return LZ4.decode(buffer);
    }
}

CompressionCodecs[CompressionTypes.LZ4] = () => LZ4Codec;

await this.consumer.run({
  eachMessage: async (res) => {
     if(res && res.topic && res.message) {
        var message = res.message;
        var key = message.key;
        var value = message.value;
     }              
  }
})
user2191611
  • 1