1

Can anyone provide me a small example in spring boot kafka where we can consume multiple topics in one single listener class.

Paizo
  • 3,986
  • 30
  • 45
Lalita
  • 23
  • 1
  • 4

2 Answers2

4

application.yml

my:
    kafka:
        conf:
            groupId: myId
            topics: topic1,topicN

you listener:

@KafkaListener(groupId = "${my.kafka.conf.groupId}", topics = "#{'${my.kafka.conf.topics}'.split(',')}")
public void storeTopicsDataToMongo(
        @Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
        @Header(required = false, name= KafkaHeaders.RECEIVED_MESSAGE_KEY) String key,
        @Payload(required = false) String record)
{
    log.trace(format("Received topic[%s] key[%s] payload[%s]", topic, key, record));
    //your code
}

or you can explore the @ConfigurationProperties and create the beans yourself, something like:

@Component
@ConfigurationProperties(prefix = "my.kafka.conf")
@Data //=> lombok
public class ConsumerConfigurationProperties {

    private String groupId;
    private List<String> topics;
}
Paizo
  • 3,986
  • 30
  • 45
  • But now I am getting the error:Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.header.Headers – Lalita Sep 20 '18 at 10:41
  • This is a problem in how you configure/build/run your application, make sure to use the same version of kafka libraries in the dependencies. Also if this answer helped you please vote up or accept it ;) – Paizo Oct 10 '18 at 13:21
  • @Paizo : If I have one topic and a group of consumers listening to this topic (say 10 of them), does it mean that 10 messages coming to this topic will be processed in parallel? – Abhishek Chatterjee Nov 09 '18 at 09:26
  • Kafka parallelism works with consumers having different group id, in spring kafka however there will be multiple thread dealing with the execution of the `@KafkaListener`. So yes performance can increase if you define the listener programatically with different group id per partition – Paizo Nov 14 '18 at 10:20
0

For consumers part of consumer group you can use following:

@KafkaListener(topics = "topic1,") public void listen(@Payload KafkaBinding record, MessageHeaders headers) throws ExecutionException, InterruptedException { ……… ……….. }

For consumers acting as assign you can use following:

@KafkaListener(id = “foo”,topicPartitions = { @TopicPartition(topic = “myTopic”,partitions = { “1” })}) public void listen(@Payload KafkaBinding record, MessageHeaders headers) throws ExecutionException, InterruptedException { ……… ……….. }

Zubair A.
  • 71
  • 3