Number of consumers in a single Kafka comsumer-group

Question

I want to consume 3 topics, with 4 partitions each, using a single consumer group.

Should the consumer group contain 3, 4, or 12 consumers?

I want to achieve optimal consumption.

Answer 1

There should be one consumer each partition for ideal consumption. So, for your case, 12 consumers should be ideal.

Answer 2

If you have N partitions, then you can have up to N consumers within the same consumer group each of which reading from a single partition. When you have less consumers than partitions, then some of the consumers will read from more than one partition. Also, if you have more consumers than partitions then some of the consumers will be inactive and will receive no messages at all.

You cannot have multiple consumers -within the same consumer group- consuming data from a single partition. Therefore, in order to consume data from the same partition using N consumers, you'd need to create N distinct consumer groups too.

Note that partitioning enhances the parallelism within a Kafka cluster. If you create thousands of consumers to consume data from only one partition, I suspect that you will lose some level of parallelism.

Answer 3

If you have 3 topics with 4 partition each. For best optimisation you should have 4 consumers per consumer group.

Reason : If you have more than 4 consumers ,your extra consumers would be left ideal, because 4 consumers will be assigned 4 partitions with 1 consumer assigned 1 partition. So in short more than 4 consumers is not required per consumer group.

If you have less consumers say 2 consumers for 4 topics , each consumer will consume messages from 2 partitions each which will overload it.

There is no limit in number of consumer groups which subscribe to a topic.