8

I want to create a concurrent @KafkaListener which can handle multiple topics each with different number of partitions.

I have noticed that Spring-Kafka only initializes one consumer per partition for the topic with most partitions.

Example: I have set concurrency to 8. I got a @KafkaListener listening to the following topics. Topic A has the most partitions - 5, so Spring-Kafka initializes 5 consumers. I expected Spring-Kafka to initialize 8 consumers, which is the maximum allowed according to my concurrency property.

  • Topic A has 5 partitions
  • Topic B has 3 partitions
  • Topic C has 1

What is the technical reason for not initializing more consumers?

How do I bypass this, such that I can initialize more consumers using the @KafkaListener annotation? (if possible at all)

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
kkflf
  • 2,435
  • 3
  • 26
  • 42

1 Answers1

8

When a listener is configured to listen to multiple topics, each consumer instance listens on all topics; Spring indeed starts 8 consumers (in this case), but the way those partitions are actually distributed across the consumers is controlled by Kafka's group management:

enter image description here

enter image description here

So you end up with 3 idle consumers in this case.

It might be possible to provide a custom partition.assignment.strategy to do the distribution the way you want, but I've never looked into that.

EDIT

I just tested with the RoundRobinAssignor...

spring.kafka.consumer.properties.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

and...

enter image description here

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Looks like the `RoundRobinAssignor` does what you want. See my edit. – Gary Russell Aug 29 '18 at 13:09
  • I have tested `RoundRobinAssignor` in my own project. It works as expected. The concurrency is utilized 100% - All the consumes are assigned partitions. – kkflf Aug 29 '18 at 13:52
  • 1
    I was stuck while trying to achieve concurrency when using a @StreamListener (or @KafkaListener) with a topic pattern - at least in some cases, a single thread/listener would service all matching topics (1 partition each), and the others had nothing to do. But configuring the RoundRobinAssignor gave me exactly what I wanted! This appears to be a critical detail when using a pattern to match topics with concurrency. Docs/examples I found mentioned the need to configure the metadata refresh with topic patterns, but I don't think I saw any references partition assignment strategy. – user1325158 May 11 '20 at 03:40