0

hi i have a consumer with multiple listeners with concurrency as 3. Each listener consume one topic. I'm trying to get the consumer lag of the containers in prometheus metric endpint. I checked the default metrics and only the listener related success and failure count and sum are available. Is there any option that i can get consumer lag exposed as a prometheus metric ?

EDIT

I'm using spring-kafka and with that in the documentation it says i can simple get the listener and broker related metrics (https://docs.spring.io/spring-kafka/reference/html/#monitoring-listener-performance). What i did was call the prometheus endpoint like myUrl/prometheus and i was able to see the listener logs.

So is there a way to view consumer lag like that?

Dushan
  • 1,365
  • 20
  • 26
  • Can you please edit your question to show your metrics endpoint output? Lag should be there – OneCricketeer Dec 17 '22 at 20:47
  • You need to get **JMX Consumer Metrics**, nothing related to Spring. Any Java Kafka consumer client has lag as JXM Bean. There's also other ways to do it, as mentioned at https://www.baeldung.com/java-kafka-consumer-lag ; but [micrometer source code](https://github.com/micrometer-metrics/micrometer/blob/main/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/kafka/KafkaConsumerMetrics.java#L103) shows lag is definitely available. – OneCricketeer Dec 19 '22 at 02:56

1 Answers1

1

As far as I know, Spring does not provide an easy way to do this (see Monitoring documentation for more details). You can use some of the existing solutions to eliminate dependency on your consumers implementations. For example:

  1. Kafka Lag Exporter.

It provides a metrics like kafka_consumergroup_group_lag with labels: cluster_name, group, topic, partition, member_host, consumer_id, client_id. It is easy to set up and can run anywhere, but it provides features to run easily on Kubernetes clusters.

  1. Kafka Exporter

It provides more different kafka metrics. For the lag, it has an example of the kafka_consumergroup_lag metric with the labels: consumergroup, partition, topic

Alexander Bobryakov
  • 131
  • 1
  • 1
  • 4