5

I'm using a PromQL query to calculate the cumulative traffic pushed/received through some interfaces on any node on the last 60 minutes. With Prometheus Node Exporter's metrics:

delta(node_network_receive_bytes_total{device=~"ens.*"}[60m])*8

And it's perfectly fine as far as the node does not reboot in that interval, the value is simply the difference between the vector's tip and its tail. When the system reboots and the counter resets, the meaning of the function stops reflecting said result.

e.g. Being this the graph for node_network_transmit_bytes_total:

IPv6 traffic

... the function will return -9MiB, instead of 10.2MiB.

I guess I can play out with rate()s to get an estimate by also using the time. But is there a better function/way to get the actual thing?

Alfageme
  • 2,075
  • 1
  • 23
  • 30

1 Answers1

9

As indicated in the documentation of delta():

delta should only be used with gauges.

You should be using the increase() function which is specific to counters.

Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for.

This is one of the main reason to distinguish between gauge and counters. See this answer about the difference.

You can identify counters by one of the following methods:

  • the type in the textfile output (ex: # TYPE http_requests_total counter)
  • the value is monotonically increasing (grafana proposes counter related function when it detects it)
  • the name should end with _total (if exporter respects best practices)
Michael Doubez
  • 5,937
  • 25
  • 39
  • This is exactly it! Next time I should try and read the docs properly... Thanks a bunch! – Alfageme Jan 09 '20 at 00:17
  • 2
    A gauge can be monotonically increasing. To distinguish for a gauge you care about the absolute value, for a counter how fast it is increasing. E.g. commits in a git repo is (usually) monotonically increasing, but it's a gauge rather than a counter. – brian-brazil Jan 09 '20 at 07:56
  • @brian-brazil Right. Someone may reset, squash and forcibly push commits sometimes, so commits may decrease. – Lewis Chan Jun 27 '21 at 13:32