Micrometer throws exception when creating a second kafka-consumer

Question

Exception occurred when upgrading to spring-boot 2.3.0. Exception is as follows:

java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'kafka_consumer_fetch_manager_records_consumed_total' containing tag keys [client_id, kafka_version, product, spring_id, topic]. The meter you are attempting to register has keys [client_id, kafka_version, product, spring_id].
    at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$17(PrometheusMeterRegistry.java:429)
    at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1932)
    at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:413)
    at io.micrometer.prometheus.PrometheusMeterRegistry.newFunctionCounter(PrometheusMeterRegistry.java:247)
    at io.micrometer.core.instrument.MeterRegistry$More.lambda$counter$1(MeterRegistry.java:884)
    at io.micrometer.core.instrument.MeterRegistry.lambda$registerMeterIfNecessary$5(MeterRegistry.java:559)
    at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:612)
    at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:566)
    at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:559)
    at io.micrometer.core.instrument.MeterRegistry.access$600(MeterRegistry.java:76)
    at io.micrometer.core.instrument.MeterRegistry$More.counter(MeterRegistry.java:884)
    at io.micrometer.core.instrument.FunctionCounter$Builder.register(FunctionCounter.java:122)
    at io.micrometer.core.instrument.binder.kafka.KafkaMetrics.registerCounter(KafkaMetrics.java:189)
    at io.micrometer.core.instrument.binder.kafka.KafkaMetrics.bindMeter(KafkaMetrics.java:174)
    at io.micrometer.core.instrument.binder.kafka.KafkaMetrics.lambda$checkAndBindMetrics$1(KafkaMetrics.java:161)
    at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
    at java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1505)
    at io.micrometer.core.instrument.binder.kafka.KafkaMetrics.checkAndBindMetrics(KafkaMetrics.java:137)
    at io.micrometer.core.instrument.binder.kafka.KafkaMetrics.bindTo(KafkaMetrics.java:93)
    at io.micrometer.core.instrument.binder.kafka.KafkaClientMetrics.bindTo(KafkaClientMetrics.java:39)
    at org.springframework.kafka.core.MicrometerConsumerListener.consumerAdded(MicrometerConsumerListener.java:74)
    at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createKafkaConsumer(DefaultKafkaConsumerFactory.java:301)
    at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createKafkaConsumer(DefaultKafkaConsumerFactory.java:242)
    at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createConsumer(DefaultKafkaConsumerFactory.java:212)
    at org.springframework.kafka.core.ConsumerFactory.createConsumer(ConsumerFactory.java:67)
    at org.springframework.kafka.core.ConsumerFactory.createConsumer(ConsumerFactory.java:54)
    at org.springframework.kafka.core.ConsumerFactory.createConsumer(ConsumerFactory.java:43)

This exception occurs when I attempt to create a consumer through ConsumerFactory.createConsumer.

There is another consumer in the app which is created by using spring-kafka through annotating the method with @KafkaListener(topics = TOPICS, groupId = GROUP_ID).

In io.micrometer.core.instrument.binder.kafka.KafkaMetrics line 146-147, I read

//Kafka has metrics with lower number of tags (e.g. with/without topic or partition tag)
//Remove meters with lower number of tags

Which means that the new metric will be discarded as it lacks the topic-tag.

Are there any reasons to why the different ways of creating consumers causes a deviation in tags? If so, is it possible to append the topic-tag to the metric created through ConsumerFactory.createConsumer?

Can you show your code/configuration? I don't see anywhere in the framework that adds those tags; standard tags are the second set - `Added: MeterId{name='kafka.consumer.fetch.manager.records.consumed.total', tags=[tag(client-id=tag-0),tag(kafka-version=2.5.0),tag(spring.id=bytesStringConsumerFactory.tag-0)]}` . So something else is adding that meter. Maybe use a debugger to figure out what's creating it? — Gary Russell, Jun 05 '20 at 14:01
This has now been registered as a issue in Micrometer: https://github.com/micrometer-metrics/micrometer/issues/2095 — Kenneth Gunnerud, Jun 08 '20 at 06:38

Kenneth Gunnerud · Accepted Answer · 2020-06-08T06:31:04.013

After some debugging, we found this:

I'll look around some more, but seems that when a consumer is started (@KafkaListener) it also adds some metics with the topic it's assigned to? Just a hypothesis so far.

Another example with less stack - Seems to register topic when scheduled task starts in KafkaMetrics.bindTo -> scheduler.scheduleAtFixedRate(() -> checkAndBindMetrics(registry), getRefreshIntervalInMillis(), getRefreshIntervalInMillis(), TimeUnit.MILLISECONDS);

Micrometer throws exception when creating a second kafka-consumer

1 Answers1