Micrometer gauge never gets updated

Question

I'm trying to build a custom gauge for micrometer which should monitor custom ThreadPoolTaskExecutor. The problem is that the gauge values are stays the same and never get updated since application start. However executer statistics (which can be seen in debugger) shows changing values.

Here is my class with gauge configuration.

@Component
public class ThreadPoolMetricProbe implements MeterBinder {

@Inject
private Executor taskExecutor;

@Override
public void bindTo(MeterRegistry meterRegistry) {
    ThreadPoolTaskExecutor exec = (ThreadPoolTaskExecutor) taskExecutor;

    long completedTaskCount = exec.getThreadPoolExecutor().getCompletedTaskCount();
    int largestPoolSize = exec.getThreadPoolExecutor().getLargestPoolSize();
    int maximumPoolSize = exec.getThreadPoolExecutor().getMaximumPoolSize();
    int poolSize = exec.getThreadPoolExecutor().getPoolSize();
    long taskCount = exec.getThreadPoolExecutor().getTaskCount();
    int activeCount = exec.getThreadPoolExecutor().getActiveCount();

    String threadNamePrefix = exec.getThreadNamePrefix();

    Gauge.builder("executorCompletedTaskCount", completedTaskCount, Double::new)
            .description(String.format("Returns the approximate total number of tasks that have completed execution of %s executor", threadNamePrefix.toLowerCase()))
            .register(meterRegistry);

    Gauge.builder("executorLargestThreadsNumber", largestPoolSize, Double::new)
            .description(String.format(" Returns the largest number of threads that have ever simultaneously been in the pool of %s executor", threadNamePrefix.toLowerCase()))
            .register(meterRegistry);

    Gauge.builder("executorMaximumAllowedThreadsNumber", maximumPoolSize, Double::new)
            .description(String.format("Returns the maximum allowed number of threads of %s executor", threadNamePrefix.toLowerCase()))
            .register(meterRegistry);

    Gauge.builder("executorCurrentNumberOfThreads", poolSize, Double::new)
            .description(String.format("Returns the current number of threads in the %s executor pool", threadNamePrefix.toLowerCase()))
            .register(meterRegistry);

    Gauge.builder("executorTotalTaskCount", taskCount, Double::new)
            .description(String.format(" Returns the approximate total number of tasks that have ever been scheduled for execution by %s executor", threadNamePrefix.toLowerCase()))
            .register(meterRegistry);

    Gauge.builder("executorActiveThreadsCount", activeCount, Double::new)
            .description(String.format("Returns the approximate number of threads that are actively executing tasks by %s executor", threadNamePrefix.toLowerCase()))
            .register(meterRegistry);


    Metrics.addRegistry(meterRegistry);

}

@Scheduled(fixedDelay = 5000)
public void measure(){
    Metrics.gauge("executorCompletedTaskCount",((ThreadPoolTaskExecutor) taskExecutor).getThreadPoolExecutor().getCompletedTaskCount());
    Metrics.gauge("executorTotalTaskCount",((ThreadPoolTaskExecutor) taskExecutor).getThreadPoolExecutor().getTaskCount());
    Metrics.gauge("executorCurrentNumberOfThreads",((ThreadPoolTaskExecutor) taskExecutor).getThreadPoolExecutor().getPoolSize());
   }
}

I'm registering here several gauges and expecting them to be updated once per request or... The method measure is mu last attempt to find correct approach to monitor and update values.

So the questions are:

1. Are gauges automatically updated? If yes - how often?
2. If gauges aren't updated automatically - then how to update them correctly?
3. Is gauge correct object for such measurements? AFAIK I cannot use counters since some of values can be decremented.
4. Do I need to do Metrics.addRegistry(meterRegistry) after all gauges are created ?

Answer 1

The thing is that Gauge.builder() method returns gauge itself, whereas meterRegistry object which is passed to

@Override
public void bindTo(MeterRegistry meterRegistry)

has method meterRegistry.gauge() can return a reference to a object-value:

AtomicDouble executorCompletedTaskCount = meterRegistry.gauge("executor.completed.task.count", new AtomicDouble(completedTaskCount));

Which then can be modified in scheduled method.

The complete code looks following:

@Component
public class ThreadPoolMetricProbe implements MeterBinder {

@Inject
private Executor taskExecutor;

private AtomicDouble executorCompletedTaskCount;
private AtomicDouble executorMaximumPoolSize;
private AtomicDouble executorLargestThreadsNumberEver;
private AtomicDouble executorCurrentNumberOfThreads;
private AtomicDouble executorTotalTaskCount;
private AtomicDouble executorActiveThreadsCount;

@Override
public void bindTo(MeterRegistry meterRegistry) {
    ThreadPoolTaskExecutor exec = (ThreadPoolTaskExecutor) taskExecutor;
    long completedTaskCount = exec.getThreadPoolExecutor().getCompletedTaskCount();
    int largestPoolSizeEver = exec.getThreadPoolExecutor().getLargestPoolSize();
    int maximumPoolSize = exec.getThreadPoolExecutor().getMaximumPoolSize();
    int currentNumberOfThreads = exec.getThreadPoolExecutor().getPoolSize();
    long totalTaskCountEver = exec.getThreadPoolExecutor().getTaskCount();
    int activeThreadsCount = exec.getThreadPoolExecutor().getActiveCount();

    executorCompletedTaskCount = meterRegistry.gauge("executor.completed.task.count", new AtomicDouble(completedTaskCount));
    executorMaximumPoolSize = meterRegistry.gauge("executor.maximum.pool.size", new AtomicDouble(maximumPoolSize));
    executorLargestThreadsNumberEver = meterRegistry.gauge("executor.largest.threads.number.ever", new AtomicDouble(largestPoolSizeEver));
    executorCurrentNumberOfThreads = meterRegistry.gauge("executor.current.number.of.threads", new AtomicDouble(currentNumberOfThreads));
    executorTotalTaskCount = meterRegistry.gauge("executor.total.tasks.ever.countunt", new AtomicDouble(totalTaskCountEver));
    executorActiveThreadsCount = meterRegistry.gauge("executor.active.threads.count", new AtomicDouble(activeThreadsCount));
    
}

@Scheduled(fixedDelay = 5000)
public void measure(){
    ThreadPoolExecutor threadPoolExecutor = ((ThreadPoolTaskExecutor) taskExecutor).getThreadPoolExecutor();
    executorCompletedTaskCount.set(threadPoolExecutor.getCompletedTaskCount());
    executorMaximumPoolSize.set(threadPoolExecutor.getMaximumPoolSize());
    executorLargestThreadsNumberEver.set(threadPoolExecutor.getLargestPoolSize());
    executorCurrentNumberOfThreads.set(threadPoolExecutor.getPoolSize());
    executorTotalTaskCount.set(threadPoolExecutor.getTaskCount());
    executorActiveThreadsCount.set(threadPoolExecutor.getActiveCount());
}

Hope it is correct way to update gauge value.

Answer 2

 meterRegistry.gauge("key_name", double)

this should give you the counter. You can increment and decrement this counter just before and after the main processing happens.

Answer 3

Try using .strongReference(true) .

Example:

Gauge.builder("executorTotalTaskCount", taskCount, Double::new)
    .strongReference(true)
    .description(String.format(" Returns the approximate total number of tasks that have ever been scheduled for execution by %s executor", threadNamePrefix.toLowerCase()))
    .register(meterRegistry);