1

I have a Kafka cluster with 40 partitions and I published a few million messages. I have a pool of consumers and for the most part the consumer lag < 100.

After my experiment, I stopped publishing any new message and left consumers to drain.

But, I don't see consumer lag come down to zero. It is struck between 0-200 for different partitions. Any reason they wont' be zero?

I am using confluent Kafka python for producer and consumer.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Icsahn
  • 11
  • 2
  • Kafka lag should become 0 eventually. How long did you give your consumers to consume everything after producer shutdown? Is your consumer shutdown managed properly? – Rishabh Sharma Aug 26 '20 at 06:58
  • Are you using transactional producers if so last transaction commit messages won’t be read by consumers there would be a lag that is equal to your partition count for each topic. So if your consumers are consuming from multiple such topics that might be the cause. – YamYamm Aug 26 '20 at 12:09
  • I didn't shut down consumers, and they have been running for >7 days now but still lag is non-zero number. I am not using transactional producers. Each commit is done async using confluent-python. – Icsahn Aug 29 '20 at 00:23

0 Answers0