4

Having a spring-boot (2.7.10) application include:

    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-core</artifactId>
    </dependency>
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-influx</artifactId>
    </dependency>

Connecting successfully to InfluxDB.

Now, executing

Metrics.counter("registration.upsigned").increment();
LOG.info("Size of registration-counter now: " + Metrics.counter("registration.upsigned").count());

Create successfully a tick in grafana

enter image description here

But the logging-result remains 0:

19-Jun-2023 07:25:15.282 INFO [https-jsse-nio2-14xx26-443-exec-123] xxx.register Size of registration-counter now: 0.0

How to keep increment the counter and surviving reboots of the application?

jccampanero
  • 50,989
  • 3
  • 20
  • 49
Grim
  • 1,938
  • 10
  • 56
  • 123

3 Answers3

1

The behavior you described could be explained by the following.

As indicated in my original answer InfluxMeterRegistry under the hood uses StepCounters to implement counters.

To obtain the actual count StepCounter uses the following code:

@Override
public double count() {
    return value.poll();
}

where value turns out to be a convenient implementation for handling double values of the StepValue class.

Please, be aware that when describing poll, they indicate:

/**
 * @return The value for the last completed interval.
 */
public V poll() {
    rollCount(clock.wallTime());
    return previous;
}

i.e. it is reporting the value for the last completed interval, not the actual one.

Consider review this test, reproduced here for completeness, I think it describes the programmed behavior in a very clear way:

@Test
void poll() {
    final AtomicLong aLong = new AtomicLong(42);
    final long stepTime = 60;


    final StepValue<Long> stepValue = new StepValue<Long>(clock, stepTime) {
        @Override
        public Supplier<Long> valueSupplier() {
            return () -> aLong.getAndSet(0);
        }


        @Override
        public Long noValue() {
            return 0L;
        }
    };


    assertThat(stepValue.poll()).isEqualTo(0L);


    clock.add(Duration.ofMillis(1));
    assertThat(stepValue.poll()).isEqualTo(0L);


    clock.add(Duration.ofMillis(59));
    assertThat(stepValue.poll()).isEqualTo(42L);


    clock.add(Duration.ofMillis(60));
    assertThat(stepValue.poll()).isEqualTo(0L);


    clock.add(Duration.ofMillis(60));
    assertThat(stepValue.poll()).isEqualTo(0L);


    aLong.set(24);
    assertThat(stepValue.poll()).isEqualTo(0L);


    clock.add(Duration.ofMillis(60));
    assertThat(stepValue.poll()).isEqualTo(24L);


    clock.add(Duration.ofMillis(stepTime / 2));
    aLong.set(27);
    assertThat(stepValue.poll()).isEqualTo(24L);


    stepValue._closingRollover();
    assertThat(stepValue.poll()).isEqualTo(27L);
}

Note that for this purpose increment is doing something "equivalent" to the different aLong.set presented in the previous test code.

I think this other test could be of help as well for describing how the value reported is the one for the last completed interval, not the actual one:

@Test
void count() {
    Counter counter = registry.counter("my.counter");


    assertThat(counter).isInstanceOf(StepCounter.class);
    assertThat(counter.count()).isEqualTo(0);
    counter.increment();
    clock.add(config.step());
    assertThat(counter.count()).isEqualTo(1);
    counter.increment();
    counter.increment(5);
    clock.add(config.step());
    assertThat(counter.count()).isEqualTo(6);
    clock.add(config.step());
    assertThat(counter.count()).isEqualTo(0);
}

In conclusion, what I tried meaning is that the log in your code fragment:

Metrics.counter("registration.upsigned").increment();
LOG.info("Size of registration-counter now: " + Metrics.counter("registration.upsigned").count());

will always print out the value for the last completed interval, not for the current one, and it may be or not the value you expect depending on whether or not the interval in which you performed the increment elapsed.

That is not specific for InfluxDB but for any Metrics registry implementation based on the step concept.

In any case, the most valuable information you can obtain is the one that is actually published to your backend, InfluxDB or the one you defined - which causes among other things the step related values to be polled.

I don't know if it will shed some additional light or not, but you can read about this rate data aggregation step based mechanism in the Micrometer documentation when they describe client side rate aggregation - please, note that InfluxBD is one of this client-side rate aggregation monitoring system. From the docs:

Micrometer efficiently maintains rate data by means of a step value that accumulates data for the current publishing interval. When the step value is polled (when publishing, for example), if the step value detects that the current interval has elapsed, it moves current data to “previous” state. This previous state is what is reported until the next time current data overwrites it. The following image shows the interaction of current and previous state, along with polling:

Behavior of a step value

* From my understanding the arrows signal publish events.

As indicated, your log (which, as you can see in the source code presented above, polls when performing count) will always report the previous value presented in the image which may or not be the same you expect according to your increments invocations and step intervals (your log may behave as the first pollAsRate presented in the image, in which it will print out 0 despite the fact two increments were performed).

If you prefer, here is a simplified version of the process, more focused in your example:

uml for process

Please, take in mind that a step based counter:

  • Always maintains two values under the hood, the current one, internal, not visible outside the class, and a previous one, visible, the one that is taking into account when publishing data.
  • Every step is configured with a time interval for rate aggregation.
  • The previous value is updated with the current value when any of two things happen:
    • A poll is requested, either by invoking count manually or usually when publishing information to your metrics registry, and the step interval is elapsed.
    • Micrometer shutdowns and claims for a closing rollover, i.e., send the last acquired data to your metrics repository.

To illustrate it, please, consider the following test case, I think it could be of help:

import java.time.Duration;

import org.junit.jupiter.api.Test;

import io.micrometer.core.instrument.Clock;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Tags;
import io.micrometer.core.instrument.simple.CountingMode;
import io.micrometer.core.instrument.simple.SimpleConfig;
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;

public class MicrometerTest {

  @Test
  public void testStepCounterBehavior() {
    Clock clock = Clock.SYSTEM;
    MeterRegistry registry = new SimpleMeterRegistry(new SimpleConfig() {
      @Override
      public String get(String key) {
        return null;
      }

      @Override
      public CountingMode mode() {
        return CountingMode.STEP;
      }

      @Override
      public Duration step() {
        return Duration.ofMillis(1000);
      }
    }, clock);

    double value = -1, previousValue = -1;
    Tags tags = Tags.empty();
    Counter counter = registry.counter("test.counter", tags);
    long start = clock.wallTime();
    counter.increment();

    int i = 0;
    while (clock.wallTime() < start + 5000) {
      value = counter.count();
      if (value != previousValue) {
        System.out.printf("Current value: %.0f. Elapsed (millis): %d.\n", value, clock.wallTime() - start);
        previousValue = value;
        if (i++ == 2) {
          counter.increment(3);
        }
      }
    }
  }
}

It will output, with some random bias, something similar to:

Current value: 0. Elapsed (millis): 0.
Current value: 1. Elapsed (millis): 601.
Current value: 0. Elapsed (millis): 1600.
Current value: 3. Elapsed (millis): 2600.
Current value: 0. Elapsed (millis): 3600.

Note that except for the first spurious output which reports a time less than the configured 1 second step interval, I think motivated by the different initialization stuff, the values and time printed are consistent with the explanation provided.

jccampanero
  • 50,989
  • 3
  • 20
  • 49
  • Hm, if I got you correctly I should call `Metrics.counter("registration.upsigned").step();` between those two lines from the question? – Grim Jul 04 '23 at 06:55
  • Sorry for the late reply. No @Grim, I think there is no such functionality: the test increases the clock using `config.step()` but only for testing purposes, you shouldn't use it, I don't know if it is even possible. I tried meaning that your log trace will always output the last interval recorded value, never the actual one. Perhaps, a way to go could be using [`LoggingMeterRegistry`](https://github.com/micrometer-metrics/micrometer/blob/main/micrometer-core/src/main/java/io/micrometer/core/instrument/logging/LoggingMeterRegistry.java): it will behave as described too, but I think it could – jccampanero Jul 04 '23 at 12:20
  • give you a proper logging. – jccampanero Jul 04 '23 at 12:20
  • Sorry I do no get the point. – Grim Jul 04 '23 at 12:53
  • My bad @Grim, I am afraid I explained myself in a confusing way, please, forget the previous comments. Please, could you review my updated answer? I hope it makes everything more clear. – jccampanero Jul 04 '23 at 17:02
  • Hm, I do not understand your diagram. Since it is not clear what `Current` represents, is the value **After the operation** meant? Could you guide me how to read this exotic diagram or use a proper UML diagram? – Grim Jul 08 '23 at 06:40
  • Sorry @Grim, is the diagram presented in the official micrometer documentation, Any way, I prepared one "UML" diagram manually. Please, could you review it? I hope it makes everything clear. – jccampanero Jul 08 '23 at 15:15
  • Ok, now I absolutly understand your point. Please let me check that! – Grim Jul 08 '23 at 16:53
  • That is great @Grim. I hope it helps. – jccampanero Jul 08 '23 at 21:57
  • I updated the answer with a small program to illustrate the described behavior. I hope it helps in some way – jccampanero Jul 09 '23 at 19:12
  • About 10 Minutes I count-up every 30 seconds. The value does not increase. – Grim Jul 10 '23 at 07:48
  • Hi @Grim. Please, can you share your configuration and the test you performed? As you can see, although I used `SimpleMeterRegistry` in the example program, the behavior should be the same for InfluxDB. the underlying step behavior is the same. – jccampanero Jul 10 '23 at 08:43
  • Hey. We could do a live-session if you like. Send me the link to a google-meet and I will join and share my screen. – Grim Jul 10 '23 at 12:36
  • I am sorry @Grim, I just realized your message. Sorry, I will be glad to help you in anything you need to, but I prefer not meeting with anybody outside of the site, I am sorry, I hope you understand. Please, can you share a simple code snippet of how you are perming the test? – jccampanero Jul 10 '23 at 16:27
  • ` /** * Test the counter for grafana. */ @Scheduled(fixedDelay = EVERY_30SECONDS) public void heartBeat() { Metrics.counter("heartBeat").increment(); LOG.info("Heartbeat hit count of " + Metrics.counter("heartBeat").count() + ". The class of the counter is: " + Metrics.counter("heartBeat").getClass().getCanonicalName()); }` I did it about 10 minutes. The metric is 0 or 2, no further incrementation. After 10 minutes: `INFORMATION: Heartbeat hit count of 2.0. The class of the counter is: io.micrometer.core.instrument.composite.CompositeCounter` – Grim Jul 11 '23 at 10:27
  • Thank you very much for sharing your test @Grim. Nice implementation by the way. Please, let me check it out: I am concerned about the reason why the counter is a composite one. – jccampanero Jul 11 '23 at 11:45
  • @Grim What logs do you obtain every 30 seconds? If you configured your step interval 1 minute and you are performing the test every 30 seconds the values `0` and `2` could be valid ones: 2 increments of 1 in a minute, interval elapsed, and reset to zero. Please, can you share the log output? – jccampanero Jul 11 '23 at 12:02
  • I shared the log-output already. The log-output does not change but repeat 20 times in 10 minutes. – Grim Jul 11 '23 at 12:31
  • I understand @Grim, thank you. – jccampanero Jul 11 '23 at 12:55
  • @Grim I think the logs still make sense: my previous comment was not right enough, I said that the test is performed every 30 seconds with a step interval of 1 minute, and that the logs report `2` for the two increments and then it reset the count to `0`, but it is not true, it reset the count to the current incremented value, which, according to your setup turns out to be `2` as well. To confirm the hypothesis please, try using the following code: – jccampanero Jul 11 '23 at 17:50
  • `/** * Test the counter for grafana. */ @Scheduled(fixedDelay = EVERY_30SECONDS) public void heartBeat() { // Consider define r as an instance variable Random r = new Random(); int amount = r.nextInt(10) + 1; Metrics.counter("heartBeat").increment(amount); LOG.info("Heartbeat hit count of " + Metrics.counter("heartBeat").count() + ". Amount incremented: " + amount + ". The class of the counter is: " + Metrics.counter("heartBeat").getClass().getCanonicalName()); }` Does the log makes more sense? – jccampanero Jul 11 '23 at 17:51
  • On the other hand, forgive me, please, I forgot that we are using Spring Boot, not only Micrometer. The logs indicates that you are using a `CompositeCounter` because Spring Boot configures Micrometer in that way in order to be able to register every `MeterRegistry` implementation you need to, mostly by introspecting your classpath. – jccampanero Jul 11 '23 at 17:53
  • @Grim Were you able to perform the suggested test? Perhaps, running it with a scheduled delay of 1 minute, the same value than your step interval could be of help as well. – jccampanero Jul 12 '23 at 21:34
  • Sorry, I just realized your answer. Does it mean that the test is still failing? – jccampanero Jul 12 '23 at 21:44
  • No, I did not executed the test. Not even once. I do not know if the test is failing or passing. A test is not allowed to change the live code. So whatever the test fail or pass, the problem will remain unchanged. Althou you have all informations yourself to find out if the test pass or fail. If you like to know if the test fail or pass, you could easily test it yourself. I see no benefit in executing your test to solve my problem. – Grim Jul 12 '23 at 21:54
  • I understand @Grim, sorry, I proposed you to perform the test because it is based in the one you provided with minimal tweaks, so I think it could be worth value to check it out because I think in fact everything is working properly and that the value `2` in your logs is just a coincidence of the combination of your increment, step interval, and fixed delay. In any way, you are right, I will try to perform the test myself. Sorry again. – jccampanero Jul 12 '23 at 22:04
  • As written in the question, there are two indicators that it is not working properly. The indicator is that neither the log nor the graph increment. Also, if the test fail or pass does not help me changing the code accordingly to increment the value correctly. If however the testresult give another idea, please post it as a answer, I will test it since you show hugh effort to solve this question. – Grim Jul 12 '23 at 22:11
  • Yes @Grim, I understand your point. I will try to review the whole setup again: in my answer, I always try describing the InfluxDB meter registry in the context of micrometer standalone, but certainly Spring Boot could be introducing a different behavior. I am sorry if I confused you in any way, it wasn't my intention. I hope to be able to come back to you again with a suitable solution in this context. – jccampanero Jul 12 '23 at 22:19
  • It is not your task, the micrometer ppl are more repsonsible I think. Let me frankly say it this way: Mind your own business. Love. – Grim Jul 12 '23 at 22:24
  • You are right @Grim, thanks. Either way, I'm sorry I couldn't help you mate. – jccampanero Jul 12 '23 at 22:28
0

Not possible at the moment (12.07.2023)

I spend about 750 bounty on this. It might be solved in the future but there was no 100% solution at the moment. It might be better to not use micrometer in Java at the moment, instead a direct influxdb connection might help.

This might change in the future, you could check other answers to this question. If there is no other answer this issue might already be solved.

Grim
  • 1,938
  • 10
  • 56
  • 123
-1

To increment and persist the counter value across application reboots, you need to store the counter value in a persistent storage like a database or a file. Since you are using InfluxDB, you can store the counter value in a separate measurement and retrieve it when your application starts :

  1. Create it in database
    A simple query will work such as : INSERT counter_values,counter_name=registration.upsigned value=0

  2. Retrieve it on app startup :

// annotation injection
@AutoWired

private InfluxDB influxDB;

private double getCounterValueFromInfluxDB(String counterName) {
    Query query = new Query("SELECT last(value) FROM counter_values WHERE counter_name = '" + counterName + "'", "your_database_name");
    QueryResult queryResult = influxDB.query(query);
    List<QueryResult.Result> results = queryResult.getResults();
    if (!results.isEmpty()) {
        List<QueryResult.Series> seriesList = results.get(0).getSeries();
        if (!seriesList.isEmpty()) {
            List<List<Object>> values = seriesList.get(0).getValues();
            if (!values.isEmpty()) {
                return Double.parseDouble(values.get(0).get(1).toString());
            }
        }
    }
    return 0;
}
  1. Init the counter using the value..
private Counter counter;

@PostConstruct
public void initCounter() {
   double initialValue = getCounterValueFromInfluxDB("registration.upsigned");
   counter = Counter.builder("registration.upsigned")
           .register(Metrics.globalRegistry)
           .increment(initialValue);
}
  1. Update the counter in db when incrementing :
private void incrementCounterAndPersist() {
   counter.increment();
   double newValue = counter.count();
   Point point = Point.measurement("counter_values")
           .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
           .tag("counter_name", "registration.upsigned")
           .addField("value", newValue)
           .build();
   influxDB.write("your_database_name", "autogen", point);
}

After that setup, the counter value should persist in database and then survive to app reboots

Ninhache
  • 26
  • 3
  • You mean `Metrics.increment` is a write-only storage? Do you mean `Metrics.counter` is designed to always return 0? Are you sure? `InfluxDB` is not a known class btw. – Grim Jun 29 '23 at 18:09
  • The first and second sentence does not make any sense, I have **not** the problem to write the value. Please see the screenshot, the value is incremented successfully correctly. Step 1 and 4 are not required as the increment does work. The step 2 and 3 are the same steps. – Grim Jun 29 '23 at 18:13