1

There is a similar question asked below:

Can a single Spring's KafkaConsumer listener listens to multiple topic?

So i now understand that i can provide an array of string to the topics parameter of the KafkaListener annotation, however i want to know the following:

  1. How do i get the topic names from a properties file as an array of strings?
  2. How does this reading from multiple topics affect the offsets? Would the clients (spring kafka) maintain offsets per topic?
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
RookieDev
  • 241
  • 3
  • 7
  • 18

2 Answers2

5

If your property is my.topics (comma-delimited)...

@KafkaListener(id = "foo", topics = "#{'${my.topics}'.split(',')}")

Yes, offsets are managed for all subscribed topics/partitions.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Hi Gary, a follow-up(I will be sure to ask this as a separate question if you think it's appropriate). If the above listener has been configured with a concurrency of say 4 and each topic has 2 partitions each, would the underlying consumer threads be allotted one each to each partition of the 2 topics. Is this how the consumer threads get distributed? – Denson Aug 08 '21 at 17:26
  • 1
    It's best not to ask unrelated questions in comments. See the note under https://docs.spring.io/spring-kafka/docs/current/reference/html/#using-ConcurrentMessageListenerContainer `When listening to multiple topics, the default partition distribution may not be what you expect. For example, if you have three topics with five partitions each and you want to use concurrency=15, you see only five active consumers, each assigned one partition from each topic, with the other 10 consumers being idle. ...` You need to change the partition assignor. – Gary Russell Aug 09 '21 at 14:20
  • I'll be happy to ask a separate question where you can answer the same comment and I will accept it. – Denson Aug 10 '21 at 04:10
1

Topic names from a properties file

@KafkaListener(topics = { "${spring.kafka.topic1}", "${spring.kafka.topic2}" })
Molay
  • 1,154
  • 2
  • 19
  • 42