Why do reading Micrometer measurement returns NaN sometimes?

Question

I'm trying to programmatically read the Meters as follows:

Get registry:

MeterRegistry registry = Metrics.globalRegistry.getRegistries().iterator().next();

Read measurement:

    double systemCpuUsage = registry.get("system.cpu.usage").gauge().measure().iterator().next().getValue();

The problem is that sometimes I get NaN.

I read about this in the docs: Why is my Gauge reporting NaN or disappearing?

but I'm not sure what I shall do. Also, I'm reading the "built-in" gauge of Spring Boot actuator (which is exposed by management.metrics.enable.process.cpu.usage=true) so I cannot change it's construction.

score 4 · Answer 1 · answered Jul 05 '19 at 17:22

4

This is due to Micrometer using 'weak references' in gauges. Since the gauge doesn't hold a strong reference to the object, when the object is garbage collected, the value becomes NaN.

If you did control the gauge creation you would want to store a reference yourself, or call strongReference(true) on the gauge.

If you are encountering with the built in Spring Boot gauges, I believe you are running into a bug. Which is very strange since the ProcessorMetrics meter binder that creates that gauge holds its own reference (though it is nullable).

Are you running on a different JVM or Runtime environment when you see the NaN?

answered Jul 05 '19 at 17:22

checketts

14,167
10
53
82

Basically, I'm seeing it when running the Spring Boot application in my IDE (IntelliJ). Is that have something to do with the problem? – IsaacLevon Jul 06 '19 at 08:57
I wonder if you are using the Spring Boot devtools and that is causing the detaching. – checketts Jul 15 '19 at 16:19
I wasn't aware of this till this moment (so answer is 'no') – IsaacLevon Jul 15 '19 at 17:12
So maybe this problem has something to do with my local setup? I don't see this problem on my production server – IsaacLevon Jul 15 '19 at 17:14

Sumeet Mondal · Accepted Answer · 2019-07-12T05:04:35.017

In this case, since you are using a "built in" metric, you can override io.micrometer.core.instrument.binder.MeterBinder#bindTo, redefine system.cpu.usage with a a custom MeterBinder implementation, and defining system.cpu.usage as (along with others which you use)

Gauge.builder("system.cpu.usage", operatingSystemBean, x -> invoke(systemCpuUsage))
.strongReference(true)//Add strong reference
.tags(tags)
.description("The recent cpu usage for the whole system")
.register(registry);

Refer io.micrometer.core.instrument.binder.system.ProcessorMetrics for example which defines it as of now.

The bean on ProcessorMetrics is defined in org.springframework.boot.actuate.autoconfigure.metrics.MetricsAutoConfiguration, you need to define your bean too somewhere. (or mark @Component)

If you don't want to rely on some predefined metric by micrometer, like to capture some custom list size, this is what you can do.

private static Map<String, Long> strongRefGauge = new ConcurrentHashMap<>();

For adding values do the following

registry.gauge("CustomListSizeGuage", getMyCustomGuageTagsFor("myListName"), strongRefGauge, g -> g.get("myListName")).put("myListName", list.size());

score 1 · Answer 3 · answered Feb 25 '22 at 14:12

just would like to mention another cause that can cause micrometer gauge to report NaN. Its in fact a default behaviour, when exception is thrown from the value function itself: https://github.com/micrometer-metrics/micrometer/blob/1.8.x/micrometer-core/src/main/java/io/micrometer/core/instrument/internal/DefaultGauge.java#L57

This exception is catched directly in gauge itself and will probably be logged in either WARN or maybe just DEBUG lvl, not sure, was not working properly in our application for some reason.

So just another possible cause worth checking :)

Why do reading Micrometer measurement returns NaN sometimes?

3 Answers3

Linked