0

For some reason, I have to use one consumer for two topics. Now the questions I have are:

  1. What is the reasoning behind we don't have a way to specify which topic we want to poll?
  2. Will a single pull/consume return messages from different topics?
  3. Assume each topic have enough messages for multiple poll requests, will each poll call return messages from different topics in round robin fashion?
  4. If the traffic of the two topics are strongly unbalanced, will the poll topic with more messages gets more attention wrt the poll request.

Every Kafka resource seems to tell me one consumer can subscribe to more than one topics but surprisingly, I can not find useful information which will answer the above questions.

dragon66
  • 2,645
  • 2
  • 21
  • 43

1 Answers1

1

  1. A consumer is considered as a subscription unit. For example, there is a configuration parameter max.poll.interval defining max period between polls before the whole consumer will be considered failed. You can manually dispatch ConsumerRecords according to its topic() value. Otherwise that's OK to create multiple consumers. You can even poll them from a single thread, just do it consciously.

  2. Yes, it will.

  3. Records would be returnd in a kind of "batched round robin" fashion. I've found that answer useful.

  4. As far as your consumer keeps up with messages being produced in total it doesn't make sense because you will get all the messages. So no extra attention could be applied. Otherwise, I suppose, but could not guarantee, you will get some constant rate from partitions you don't keep up with, and lower rate from other ones. If it ends up you don't keep up with all partitions you will get constant rate from all of them.

Hope it helped.

gdomo
  • 1,650
  • 1
  • 9
  • 18