1

I have Three topics each having three partitions on a cluster of kafka.

now, there are total 9 partitions. and when i create 9 consumers... the 6 are being idle. only three consumers are being used.

the expectation is: each consumer should pickup one partitions and hence, 9 consumer should pick up documents from 9 partitions

but what happens is: one consumer picks up messages from three paritions one of different topic.

e.g. i have three topics Topic_A,Topic_B and Topic_C and three partitions each. hence parititions are as below:

Topic_A_0, Topic_A_1, Topic_A_2, Topic_B_0, Topic_B_1, Topic_B_2, Topic_C_0, Topic_C_1, Topic_C_2

When i create 9 consumers, the distribution works as below:

Consumer1: Topic_A_0,Topic_B_0,Topic_C_0 
Consumer2: Topic_A_1,Topic_B_1,Topic_C_1 
Consumer3: Topic_A_2,Topic_B_2,Topic_C_2
Consumer4,Consumer5,Consumer6,Consumer7,Consumer8,Consumer9 are idle

It should be 

Consumer1: Topic_A_0
Consumer2: Topic_A_1
Consumer3: Topic_A_2

Consumer4: Topic_B_0
Consumer5: Topic_B_1
Consumer6: Topic_B_2

Consumer7: Topic_C_0 
Consumer8: Topic_C_1 
Consumer9: Topic_C_2

Is there any configuration i need to let all 9 consumer pick up messages from 9 unique parititons?

Marius Waldal
  • 9,537
  • 4
  • 30
  • 44
Radhi
  • 6,289
  • 15
  • 47
  • 68
  • Are all consumers subscribed to all three topics using the `subscribe()` API? Are you using the default partition assignment strategy or using something specific in the config? – vahid Jul 11 '18 at 05:56

2 Answers2

0

Make sure your all your consumers are subscribing to same set of topics under the same consumer group id. For the list of topics, you can pass a predefined list or a regular expression for consumers to subscribe from. The consumer-id can be set using group.id property in consumer.

Bitswazsky
  • 4,242
  • 3
  • 29
  • 58
0

The default partition assignment strategy doesn't work across topics so this is the expected behaviour. A similar question here : Kafka Consumers are balanced across topics

ppatierno
  • 9,431
  • 1
  • 30
  • 45