Apacke Kafka Listener Threads

Question

Let's say I have 50 Kafka Topics, each with 3 partitions, that's a total of 150 partitions. If I have a KafkaListener/Consumer configured for each of these 150 partitions (due to high volume on each partition), it means I have 150 listeners running. From what I understand, each listener gets its own thread. So does that mean there will be 150 active threads in this scenario? That seems like a lot. Is there any way I could restrict this to a max number of threads at a time(say, 20)?

Answer 1

Kafka always gives a single partition’s data to one consumer thread

For restricting the thread count to 20 with 150 partitions, you could set the concurrency to 8 which should effectively create 8 separate consumer instances & limit the thread count to a maximum of <19 (150/8).

This is another stack post on the topic which explains it a bit more for setting concurrency in your kafka listener factory. Also this is another decent source for explaining some of the concepts around kafka topics & partitions.

Answer 2

1. If you want to maximize consumption, yes, you need 150 consumer threads on 150 partitions (1 consumer thread per partition). In case you only have 20, you can't want to maximize consumption, instead, now 1 consumer is responsible for consuming for example 150/20=7 partitions. It's a tradeoff.

2. You won't want your app consumes 50 topics at a time. It leads to the scale problem. Let say now one topic needs to increase the number of its partitions from 3 to 6, you horizontally scale your app from 1 to 2 instances, which means from 150 consumers to 300 consumers and there are only 150 + (6-3) = 153 active consumers, the others become idle.