0

I know in KafkaJS we can consume message with the following code

const run = async () => {
    await kafkaClient.consumer.subscribe({ topic: 'mytopic', fromBeginning: true })
    await kafkaClient.consumer.run({
        eachBatchAutoResolve: false,
        eachBatch: async ({ batch, resolveOffset, heartbeat, isRunning, isStale }) => {
            for (let message of batch.messages) {
                if (!isRunning() || isStale()) break
                processMessage(message)
                resolveOffset(message.offset)
            }
        },
    })
}

And the message format is like this

{
  magicByte: 2,
  attributes: 0,
  timestamp: '540669',
  offset: '601953',
  key: <Buffer 39 63 37 23>,
  value: <Buffer 7b 65 32 65 37 38 ... 555 more bytes>,
  headers: {
    'myheader': <Buffer 61  6f>,
  },
  ‧‧‧‧‧‧
}

But now I'm using KafkaListener in spring boot as consumer.

    @org.springframework.kafka.annotation.KafkaListener(
            topics = "mykafkatopic",
            groupId = "groupId"
    )
    void listener(WhatTypeShouldThisBe data){
        
        }
    }

In JS, the type of the message is object, so I'm confused about what type of object should be used in the java listener function to receive the message?

And the messages produced by KafkaJS are serialized, how to deal with the serialization in java consumer?

Update1: Should I define a class like this

public class messageFormat{
        int magicBytes;
        ‧‧‧
        byte[] key;
        selfDefinedHeaderClass header;
}

And then modify the ConsumerFactory function to this

public ConsumerFactory<String, messageFormat> consumerFactory() {
    // ...
    return new DefaultKafkaConsumerFactory<>(
      props,
      new StringDeserializer(), 
      new JsonDeserializer<>(messageFormat.class));
}
John Smith
  • 1
  • 2

1 Answers1

0

You need to define a Java class with the fields from the JSON and use the JsonDeserializer in the consumer configuration, configured to deserialize the JSON and create an instance of that class.

https://docs.spring.io/spring-kafka/docs/current/reference/html/#json-serde

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Please refer to my edited question(`Update1`). Do you mean I have to do like that? – John Smith Jun 01 '22 at 18:11
  • I see you are new here; don't put code in comments, it is barely readable; edit the question and comment that you have done so. Yes, something like that, or you can simply consume it as a `Map`. However `` does not look like valid JSON to me, so if that's really the format (rather than, say a Base64 string), you will probably need a custom deserializer. – Gary Russell Jun 01 '22 at 18:15
  • `` is the printed result by running `console.log(message)` in js. I think it should be the byte array encoded with utf8? – John Smith Jun 01 '22 at 18:19
  • Binary data is usually encoded as a Base64 String. It would be useful to see what one of these objects looks like, using the `kafka_console_consumer.sh`. – Gary Russell Jun 01 '22 at 18:38