How to publish the messages to two topics from single producer and consuming in single listener in Kafka

Question

I am Trying to publish messages two different topics from single producer.

here i created two topics:

   @Bean
    public NewTopic multi1() {
        return TopicBuilder.name("multi1").partitions(1).build();
    }
    @Bean
    public NewTopic multi2() {
        return TopicBuilder.name("multi2").partitions(1).build();
    }

this is how iam sending the messages to two topics:

public void sendingtomultitopic()
{
    IntStream.range(0, 100).forEach(i->this.template.send("multi1", "mutli1 data value->"+i));      
    IntStream.range(0, 100).forEach(i->this.template.send("multi2", "multi2 data value->"+i));
    logger.info("sending finished");
}

is above method is correct way to send to multiple topics?

here below iam trying to consume messages

@KafkaListener(topics = {"multi1,multi2"}, groupId = "diffgroupid3")
    public void consumingfrommultitopics(String data) {
        logger.info(String.format("consumingfromtwotopics -> %s", data));
    }

exception iam receiving:

org.apache.kafka.common.errors.InvalidTopicException: Invalid topics: [multi1,multi2]

iam able to the published data in those two topics.

but this consumer is not retrieving any messages, please help me here?

try defining the multiple topic names like ``topics = {"multi1","multi2"}`` — pratap, May 31 '21 at 12:01
is that the correct way to publish the messages to multiple topics — Bharath, May 31 '21 at 13:24

score 2 · Answer 1 · answered May 31 '21 at 12:25

You can use from config (application.yml)

test:
  kafka:
    sender:
      serverAddress: xxxxx:port
    topic:
      consumer: topic1,topic2

@KafkaListener(topics = "#{'${test.kafka.topic}'.split(',')}"
  public void consumingfrommultitopics(String data) {
        logger.info(String.format("consumingfromtwotopics -> %s", data));
    }

score 2 · Answer 2 · answered May 31 '21 at 12:38

You have a typo in 'topics' form, should be

@KafkaListener(topics = {"multi1","multi2"}...

(check out Can a single Spring's KafkaConsumer listener listens to multiple topic?)

You can also have two separate @KafkaListener, each for every topic. What works best for your use-case.

How to publish the messages to two topics from single producer and consuming in single listener in Kafka

2 Answers2