How to increase Kafka Consumers no

Question

Say for example - Kafka topic name - topic_X topic_X has 500 partitions in the Kafka Broker

Now we have set 500 consumers for the Consumer_X group to process each partitions.

where to Run these 500 consumers ? In a single machine 500 thread ? Is this possible - because thread vs core has a relationship

How to achieve this ?

Even we reduce to 100 consumers in Consumer_X group (1 consumer =5 partitions) then also same above questions valid

Please explain this

Answer 1

The main purpose of a consumer group is to split the work across multiple machines. Running one consumer per thread is also possible but in a high load topic scenario, it doesn't bring any value.

Polling records from Kafka is highly efficient. Usually, processing them is the bottleneck. So, the main idea is to split that work across multiple machines so that every consumer application can keep up with the workload.

Answer 2

Having a 1:1 relationship between partitions and consumers is only a thing if in your use case:

1. Records are evenly spread: that means that your producer has to have a minimal control over which partitions are being written when records are sent. After Kafka 2.4 this means a lot because the default partitioner is no longer the RoundRobin but the Sticky one. So you'd need to explicit set the RoundRobin in the producer to make this happen.
2. Even partition assignment: The default behavior of how partitions are assigned to consumers has changed in the recent versions of Kafka. After the introduction of the incremental/cooperative rebalance protocol there is a trend where partitions are assigned to the same consumers alive to reduce the stop-the-world pauses during rebalancing. Using an more even spread of assignment means giving up of the innovations of the new protocol and thus, making your consumers more bound to pause if liveness of one of them is compromised.

Unless you're dealing with a high throughput use case where each record need to be processed ASAP, having a 1:1 between partitions and consumers is too costly since each consumer thread doesn't come for free. Putting them in the same box for instance is not recommended since you may have few cores available and a higher # of threads will cause constant context switching reducing throughput. The solution would be spreading those threads over multiple boxes but them... here comes the cost problem again.

I would measure how efficient is having 100:1. This seems to be reasonable, especially if the ingress throughput is not high and some consumer lag is tolerable.