3

I have been playing around with Kafka for a few months now and I have noticed that I have a rather large number of message/sec under the internal Kafka topic called __consumer_offsets. Now we have a hand full of other topics that are rather small in terms of the number of message across the brokers. At the most we can see ~1k message/sec on a couple topics, but I almost always see 15-20k message/sec from _consumer_offsets. Can someone point me to something that will explain this or provide some insight in why the consumer_offsets topic has so many messages.

enter image description here

Thanks

unreal
  • 53
  • 6
  • it holds info about offset of each topic/partition/consumer/consumergroup so probably thats why its pretty big :) – hi_my_name_is Mar 20 '18 at 08:04
  • 1
    Thanks Robin and freakman. I was kind of thinking this but I was having a hard time believing the number of messages were that high for this topic considering my topics don't seem to see nearly as many messages. – unreal Mar 20 '18 at 16:41

2 Answers2

1

This is how Apache Kafka works. See http://kafka.apache.org/documentation.html#impl_offsettracking for more details.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
0

I have The same. Number of messages in all other topic is insignificant in comparison to __consumer_offsets topic. Is it problem because high number of commits by producers ? If not, we should hide this topic from monitoring graph for clear seeing another topics.

Cloud66
  • 31
  • 1
  • 3