0

I am creating 6 consumers with same group-id. All the consumers are subscribing to 2 topics with 3 partitions each. Since there are 6 consumers and 6 partitions across 2 topics I am expecting all the consumers to be used. But I don't see all the consumers being used, is there a way I can force it to re balance ?

I am using kafka 0.10.2.0

Naga
  • 33
  • 5

1 Answers1

2

Assigning partitions to consumers in the same consumer group is not done across topics. Following what is happening to you ... You have t1_p0, t1_p1 and t1_p2 (partitions on topic1) and t2_p0, t2_p1 and t2_p2 (partitions on topic2). Then you have a consumer group with 6 consumers c1, c2, ... c6. When they subscribe to topic1, it can happen that c1, c2 and c3 get t1_p0, t1_p1 and t1_p2; the other 3 consumers (c4, c5, c6) are idle because there aren't enough partitions. When the same consumers subscribe to topic2, it can happen the same: c1, c2 and c3 get t1_p0, t1_p1 and t1_p2; the other 3 consumers (c4, c5, c6) are idle because there aren't enough partitions. I know that you would like having c4, c5 and c6 getting these partitions from topic 2 but as I said partitions assignment doesn't happen across topics but for partitions in the same topic. In this scenario you have c4, c5 and c6 in idle and not used.

ppatierno
  • 9,431
  • 1
  • 30
  • 45
  • Thank You....Since consumer config has group Id param and can subscribe to multiple topics, I was assuming consumers in the group can consume from partitions across topics.. I changed it to have different groups for different topics and I see them are balanced – Naga Jul 13 '18 at 16:36