How to expose kafka metrics to /actuator/metrics with spring boot 2

Question

I was looking a while and didn't seem to find the answer. I'm using Spring boot 2, Spring Kafka 2.1.4 and I want to see the kafka consumer metrics in the /metrics endpoint of spring boot actuator. What I don't understand is - should I implenemt the exposure myself or is this comes out-of-box in boot 2 ?

If I to implement this my self, what is the best way to do it?

Kafka exposes almost the same metrics through JMX, so I suggest you to create a custom `MeterBinder` to expose these metrics. Plus, you can create spring boot `autoconfigure and starter modules` to simplify the usage. You can find an example (ready to use) here: https://github.com/luiztoscano/spring-boot-kmetrics Hope it helps. — Luiz Toscano, Jan 01 '19 at 23:27

score 5 · Answer 1 · answered Sep 25 '18 at 23:22

5

Targeted for micrometer v1.1.0 is the KafkaConsumerMetrics implementation of MeterBinder. This should expose the kafka consumer metrics you're looking for.

Source Reference:

https://github.com/micrometer-metrics/micrometer/blob/master/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/kafka/KafkaConsumerMetrics.java

answered Sep 25 '18 at 23:22

Derek Slife

20,750
1
18
22

2

Is there an equivalent KafkaProducerMetrics available? – Vassilis Oct 22 '18 at 15:31

score 2 · Answer 2 · answered Sep 11 '20 at 03:42

2

add this config works for me:

spring.jmx.enabled=true

answered Sep 11 '20 at 03:42

ThinkAlone

51
1

This might be the correct answer for the time of writing this. It worked for me. – Anakin001 Sep 30 '20 at 09:41

score 1 · Answer 3 · answered Mar 27 '18 at 15:07

1

You are right, there is no out-of-the-box one. You don't have choice unless to implement your own MeterBinder. For the Apache Kafka Consumer metrics per se, you should inject a KafkaListenerEndpointRegistry and call its getListenerContainers() and use their metrics() to bind to the provided MeterRegistry.

When you come up with something, feel free to contribute the solution back to Spring Boot.

answered Mar 27 '18 at 15:07

Artem Bilan

113,505
11
91
118

The problem seems to be that there is no elegant way to migrate the KafkaMetric objects to Micrometer metrics. The only way I see is looping all kafka metrics and creating Micrometer identical metric for each of them – Michael Azimov Mar 29 '18 at 08:25
@MichaelAzimov I am looking to move the KafkaStreams metrics to micrometer. How did you proceed for the conversion ? – Guillaume Jobin Dec 05 '18 at 04:21

score 1 · Answer 4 · answered Dec 13 '22 at 08:02

This works for me(For Spring Boot 2.3.0.RELEASE and above):

@Autowired
private MeterRegistry meterRegistry;

@Bean
public ConsumerFactory<Object, Object> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
    props.put(ErrorHandlingDeserializer.KEY_DESERIALIZER_CLASS, JsonDeserializer.class);
    props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, JsonDeserializer.class);

    //More code here

    DefaultKafkaConsumerFactory<Object,Object> cf = new DefaultKafkaConsumerFactory<>(props);
    cf.addListener(new MicrometerConsumerListener<>(meterRegistry));
        
    return new DefaultKafkaConsumerFactory<>(props);
}

build.gradle
implementation 'io.micrometer:micrometer-registry-prometheus'

Metrics will be available at /actuator/prometheus/kafka_consumer_<metric-name>.

How to expose kafka metrics to /actuator/metrics with spring boot 2

4 Answers4

Linked

Related