Micrometer-springboot: Hikaricp , Tomcat and jdbc metrics are not exported to DataDog

Question

Hikaricp , Tomcat and jdbc metrics are not being exported to DataDog

we have setup springboot app to push the metrics to datadoghq, it does export 60 metrics, however the metrics like hikaricp, tomcat and jdbc are missing.

hikaricp, tomcat and jdbc - these mertics are listed under /actuator/metrics endpoint, but not exported to datadog.

springBootVersion = '2.3.3.RELEASE'
springCloudVersion = 'Hoxton.SR7'
implementation 'io.micrometer:micrometer-registry-datadog:latest.release'

Is there any additional settings required to push hikaricp, tomcat and jdbc metrics ?

I found the issue, I have another LoggingMeterRegistry as primary, that was stopping additional metrics to be exported to datadogmeter registry. If I make DatadogMeterRegistry as the primary one, LoggingMeterRegistry do not log tomacat, hikari and jdbc metrics. is there any way I can export all metrics and log to console using LoggingMeterRegistry — Suresh Naik, Sep 09 '20 at 17:33
Don't make either a primary. By default a 'CompositeRegistry' should be the primary and the other two will be delegated to. — checketts, Sep 10 '20 at 16:25
if I don,t make it one as primary, it complains that there are two qualifying beans: ```Parameter 0 of method webMvcMetricsFilter in org.springframework.boot.actuate.autoconfigure.metrics.web.servlet.WebMvcMetricsAutoConfiguration required a single bean, but 2 were found: - getDatadogMeterRegistry: - loggingMeterRegistry:``` — Suresh Naik, Sep 10 '20 at 17:19
That is strange since actuator include `CompositeMeterRegistryConfiguration` which is autoconfigured. You can manually create a composite registry and manually add your other registries to it if you like. (I see you did that in your answer) — checketts, Sep 10 '20 at 21:04

score 3 · Answer 1 · answered Sep 09 '20 at 15:41

3

Usually metrics exposed to /actuator/metrics are sent to the metrics system like datadog.

You can try to check what exactly gets sent to datadog by examining the source code of DatadogMeterRegistry

Put a breakpoint in the publish method and see what gets sent, or, alternatively set the logger of the class to "trace" so that it will print the information that gets sent to the datadog (line 131 in the linked source code).

Another possible direction to check is usage of filters (see MeterFilter) that can filter out some metrics.

answered Sep 09 '20 at 15:41

Mark Bramnik

39,963
4
57
97

Thanks for responding, yes I did inspect and also enabled TRACE, it did send only 60 metrics. I found the issue, I have another LoggingMeterRegistry as primary, that was stopping additional metrics to be exported to datadogmeter registry. – Suresh Naik Sep 09 '20 at 17:01
If I make DatadogMeterRegistry as the primary one, LoggingMeterRegistry do not log tomacat, hikari and jdbc metrics. is there any way I can export all metrics and log to console using LoggingMeterRegistry ? – Suresh Naik Sep 09 '20 at 17:32
It sounds like the registries are not initialized properly... Aren't you relying on spring boot and only provide dependencies and letting spring boot to configure everything for you? What is primary registry? Could you llease share the configurarion? In general micrometer uses CompositeMeterRegistry to handle the situation with multiple registries, so you won'r need to use primary I guess... – Mark Bramnik Sep 10 '20 at 03:26
If I dont make one of as primary the startup fails with this error ```Parameter 0 of method webMvcMetricsFilter in org.springframework.boot.actuate.autoconfigure.metrics.web.servlet.WebMvcMetricsAutoConfiguration required a single bean, but 2 were found: - getDatadogMeterRegistry: - loggingMeterRegistry:```. – Suresh Naik Sep 10 '20 at 17:21
1

You should share the initialization code. In general I believe you should go with Composite Registry as a "primary" one, It will include both Logging and DataDog Registries inside, but from the "whole system" in general the CompositeRegistry is the one to work with. – Mark Bramnik Sep 10 '20 at 17:23
I have configuration like this: ```@Bean @Primary public DatadogMeterRegistry getDatadogMeterRegistry(DatadogConfig config) { return DatadogMeterRegistry.builder(config).build(); }``` ``` @Bean public LoggingMeterRegistry loggingMeterRegistry() { LoggingRegistryConfig config = new LoggingRegistryConfig() { .... return LoggingMeterRegistry.builder(config).build(); }``` – Suresh Naik Sep 10 '20 at 17:23
and in gradle I have this dependency : ```implementation 'io.micrometer:micrometer-registry-datadog:latest.release'``` – Suresh Naik Sep 10 '20 at 17:25
Yes, but why do you have it? Spring boot should "recognize" all the registries automatically (it should be autonfiguration modules with spring.factories and everything). And then it should create a composite registry so that you only need to add the relevant maven dependencies – Mark Bramnik Sep 10 '20 at 17:25
I have added LoggingMeterRegistry to log the metrics, without that explicit bean I couldn't log the metrics to console log. not sure if any other way to log without explicitly declaring the bean – Suresh Naik Sep 10 '20 at 17:32
1

Maybe it doens't have autoconfiguration, I don't really remember... Anyway you should try to use CompositeMeterRegistry as a "primary" registry and LoggingMeterRegistry should be added to the composite with the help of 'add' method (see documentation https://javadoc.io/static/io.micrometer/micrometer-core/1.0.3/index.html?io/micrometer/core/instrument/composite/CompositeMeterRegistry.html) – Mark Bramnik Sep 10 '20 at 17:53
Thank you Mark, that did the trick.. thanks a lot!! – Suresh Naik Sep 10 '20 at 18:35

score 3 · Accepted Answer · edited Sep 10 '20 at 19:25

This did the trick : Thanks to @MarkBramnik

    @Bean
    @Primary
    CompositeMeterRegistry compositeMeterRegistry(DatadogMeterRegistry datadogMeterRegistry, LoggingMeterRegistry loggingMeterRegistry) {
        CompositeMeterRegistry compositeMeterRegistry = new CompositeMeterRegistry();
        compositeMeterRegistry.add(datadogMeterRegistry);
        compositeMeterRegistry.add(loggingMeterRegistry);
        return compositeMeterRegistry;
    }

Micrometer-springboot: Hikaricp , Tomcat and jdbc metrics are not exported to DataDog

2 Answers2