Confusion about micrometer metrics - Isn't the gauge supposed to calculate the value automatically just before it is submitted?

Question

I am exploring micrometer and aws cloudwatch.
I think there is some understanding gap -
I've create a gauge which is supposed to return the number of connections being used in a connection pool.

public MetricService(CloudWatchConfig config) {
    this.cloudwatchMeterRegistry = new CloudWatchMeterRegistry(config, Clock.SYSTEM, CloudWatchAsyncClient.create());

    gauge = Gauge.builder("ConnectionPoolGauge", this.connectionPool, value -> {
                Double usedConnections = 0.0;
                for (Map.Entry<String, Boolean> entry : value.entrySet()) {
                    if (entry.getValue().equals(Boolean.FALSE)) {
                        usedConnections++;
                    }
                }
                return usedConnections;
            })
            .tag("GaugeName", "Bhushan's Gauge")
            .strongReference(true)
            .baseUnit("UsedConnections")
            .description("Gauge to monitor connection pool")
            .register(Metrics.globalRegistry);

    Metrics.addRegistry(cloudwatchMeterRegistry);
}

As you can see, I am currently initiating this gauge in a constructor. Passing the connectionPool instance from outside.
Following is a controller method which consumes the connection -

@GetMapping("/hello")
        public String hello() {
    // connectionPool.consumeConnection();
    // finally { connectionPool.releaseConnection();}
    }

Step interval is set to 10 seconds. My understanding is - Every 10 seconds, Micrometer should automatically execute the double function passed to the gauge.
Obviously, it is not happening.
I've seen some code samples here which are explicitly setting the gauge value (in a separate thread or scheduled logic).

I also tried with a counter which is instantiated only once, but I explicitly invoke the increment method per call to hello method. My expectation was this counter would keep on incrementing, but after a while, it drops to 0 and starts counting again.
I am totally confused. Appreciate if someone can put light on this concept.

Edit: Tried following approach for creating Gauge - still no luck.

cloudwatchMeterRegistry.gauge("ConnectionPoolGauge", this.connectionPool, value -> {
    Double usedConnections = 0.0;
    System.out.println("Inside Guage Value function." + value.entrySet());
    for (Map.Entry<String, Boolean> entry : value.entrySet()) {
        if (entry.getValue().equals(Boolean.FALSE)) {
            usedConnections++;
        }
    }
    return usedConnections;
});

This doesn't return the instance of Gauge, so I cannot call value() on it. Also the gauge is not visible in AWS Cloudwatch. I can see the counter in cloudwatch that I created in the same program.

Answer 1

Micrometer takes the stance that gauges should be sampled and not be set, so there is no information about what might have occurred between samples. After all, any intermediate values set on a gauge are lost by the time the gauge value is reported to a metrics backend anyway, so there seems to be little value in setting those intermediate values in the first place.

If it helps, think of a Gauge as a "heisen-gauge" - a meter that only changes when it is observed. Every other meter type provided out-of-the-box accumulates intermediate counts toward the point where the data is sent to the metrics backend.

So the gauge is updated when the metrics are published, here are a few tips to troubleshooting this:

1. Put a brake point in the publish method of your CloudWatchMeterRegistry and see if it is called or not.
2. You are using the Global registry (Metrics.addRegistry) as well as keeping the reference to CloudWatchMeterRegistry (this.cloudwatchMeterRegistry = new CloudWatchMeterRegistry). You don't need both, I would suggest to do not use the Global registry and inject the registry you have wherever you need it.
3. I'm not sure what you are doing with the connection pool (did you implement your own one?) but there is out-of-the-box support for HikariCP and DBCP is publishing JMX counters that you can bind to Micrometer.