0

In a scenario, where I have N consumers (all consumers have 1 stream/no partitioning), each subscribed to separate set of topics, how to process new topics, added by producers?

Should I create new consumer to each added topic? Or can I add this topic to already working consumer? (how to do that?)

Or better keep 1 consumer group with N consumers and not divide topics between N consumers(streams)?

user3391013
  • 17
  • 1
  • 6
Vladimir Nabokov
  • 1,797
  • 1
  • 14
  • 19

1 Answers1

1

The initial config, as described, where topics devided between consumers of the same consumer group is not goood at all. Consumer will not be able to fail=over each other, because they have different lists of topics. Kafka will issue unnecessary rebalance.

You need not add consumer for each added topic - you better consume all such topics from beginning, adding them to common path like /MYTOPICS/*

You better divide topics by partitions and add new partitions if you need. This will be transparent and without effort.

Vladimir Nabokov
  • 1,797
  • 1
  • 14
  • 19