I am starting with kafka , have a question on the consumer groups. We have an application where we want different consumers from same group subscribing to different topics. The grouping is done based on some business criteria . To be specific consumer 1 from group A and consumer 2 from group A are subscribed to Topic 1 and Topic 2 each with 10 partitions. Does this mean that there consumer 1 can scale to 10 and consumer 2 also can scale to 10 since they are subscribed to different topics. Is this a correct design
Asked
Active
Viewed 854 times
1
-
it depends on how your code is, need to look at code how you are spinning consumers – Ryuzaki L Apr 14 '21 at 19:53
1 Answers
1
yes, since within a topic kafka try to assign partitions to consumers as equal as possible. The key is topic:consumer_group_id
so it doesn't matter another_topic:same_consumer_group_id
- it's another key, and consumer with topic:consumer_group_id
could be scaled to partitions number

J.J. Beam
- 2,612
- 2
- 26
- 55