2

I'm using JMX to gather metrics from Apache Kafka and send to Librato, a visualization and monitoring service, via collectd. The problem is that some metrics seems to be reporting wrongly. For instance, without anyone using the Kafka cluster, some nodes report high number of incoming messages for minute (like 15,000), whereas the others report 0, as expected.

Here is one of the metrics configuration in collectd:

<MBean "kafka-all-messages">
  ObjectName "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec"
  InstancePrefix "all"
  <Value>
    InstancePrefix "kafka-messages-in"
    Type "counter"
    Table false
    Attribute "MeanRate"
  </Value>
</MBean>

And here is the graph in Librato:

Messages per second visualization in Librato

Does anyone know what's going wrong? Is it my collectd configuration, such as Type or anything like this?

Matheus Portela
  • 181
  • 1
  • 9

1 Answers1

1

Just a small follow up on this issue: it wasn't neither JMX nor collectd mistakes. I verified JMX, collectd and Librato were with consistent values by logging in directly to each Kafka broker and reading the values in their MBeans.

The problem was we were using Kafka clients with too small auto-commit interval, which generated huge numbers of messages being sent to each Kafka broker.

Matheus Portela
  • 181
  • 1
  • 9