Pushing metrics to pushgateway replaces the old one

Question

We have a java application called MonitorApp that is running in a docker container. The purpose of this app is to perform certain tasks, push the metrics to the Pushgateway and die.

This MonitorApp runs as a Kubernetes Cron Job. This means after publishing matrices, it will get killed and a new instance will come up after some time.

I am using Java Pushgateway APIs to publish the matrices. Here is my sample code for the same:

class MonitorApp {

   public void static main(String args[]) {
       
        List<String> labelKeys = Arrays.asList("key01", "key02");
        List<String> labelValues = Arrays.asList("value01", "value01");

        PushGateway client = new PushGateway("localhost:9091");
        CollectorRegistry registry = CollectorRegistry.defaultRegistry;

        String jobName = "midm_monitor_app";
        String metric = "metrin_name";
        String help = "metric_help";


        Counter counter = Counter.build()
                .name(metric)
                .help(help)
                .labelNames(labelKeys.toArray(new String[0]))
                .register(registry);
        counter.labels(labelValues.toArray(new String[0])).inc();

        client.push(registry,jobName);
     }
}

The issue is that I am not able to increase the value of the Counter metric. I am not able to find a way to fetch the previous counter value so that I can increment it.

I have read that purpose of the Pushgateway is to handle this kind of use case.

Please suggest how to should I handle this?

score 1 · Answer 1 · answered Aug 12 '21 at 12:39

From what I understand, you want a metric to be increased between multiple run of your instance. This is not the goal of the push gateway: it will only store the values you push - not aggregate them in any way except for add/replace of metrics group.

Although I have never used them, there are solutions to have aggregation using other monitoring stacks (like statd) or specific projects like pushprom. You could also push runs as events (in elasticsearch by example) for further aggregation.

If it is really a sensitive part of your application monitoring, I advise you stay away from SPOF solutions. You could roll your own metric cache leveraging any (distributed) key/value store at your disposal.

Pushing metrics to pushgateway replaces the old one

1 Answers1