rrd_fetch does not contain values from last update. why?

Question

I am storing information from temperature sensors in a rrd. I want to retrieve the values from a given timespan in php.

For data from the last 2 hours, I use this:

rrd_fetch($file, array("AVERAGE", "-s -2h"));

This is working. However, the last value that I can get via rrd_lastupdate($file); is not found in the data from rrd_fetch. How can I adapt my code so that the rrd_fetch call also contains data from the last update?

The rrd is setup like this in Python:

   rrdtool.create(
        RRD_FILE,
        "--start", "now",
        "--step", "300",
        "RRA:AVERAGE:0.5:1:288",
        "RRA:AVERAGE:0.5:12:336",
        "RRA:AVERAGE:0.5:288:365",
        "DS:temp:GAUGE:600:0:50")

And updated like this in a while loop:

while True:
    temp = get_value()
    data_rrd = "N:{0}".format(temp)
    try:
        rrdtool.update(file, data_rrd)
    except:
        print("Failed to write to RRD.")
    time.sleep(300)

Answer 1

This is because lastupdate() and fetch() do different things.

When you add data to an RRD, there is a multi-stage process going on in the background. First, the data are turned to Rates; then Normalised; then Consolodated.

1. Data submitted update()

2. Data are converted to a Rate. If your DS is of type gauge then this is a noop, but other types will be converted based on the time of the previous update.

3. Now we get the value returned by lastupdate(), which is this DP (datapoint)

4. Data are Normalised. This means that we assume a linear progression from the last known datapoint, and calculate the average over the last step Interval (300s in your example above). This gives a new PDP (primary datapoint) on a time window boundary (IE with the PDP time being a multiple of the Interval)

5. The PDP are now Consolodated. When sufficient have been collected,a new CDP (Consolodated Datapoint) is written to the RRA, depending on the CF and XFF. If your RRA has 1cdp=1pdp (as in your case) then the CDP should be the same as the PDP.

6. Now we get the CDP in the RRA, which is what fetch() returns.

From this, you can see that the value you store is not the same as the lastupdate() (due to Normalisation and Rate); and this is not the same as the fetch() (due to Consolodation).

In addition, you won't see the value appearing in the lastupdate() until the time window has completed; and you wont see it appearing in fetch() until the RRA CDP is completed.

Since your fetch() is not asking for more data than in your 1cdp=1pdp RRA, you'll avoid changes due to consolodation, but you're likely hitting Normalisation.

If you want the shown values to be the same, then you need to ensure all 3 operations are no-ops.

1. Use type gauge to stop Rate conversion

2. Ensure all updates are done exactly on a time boundary to avoid Normalisation. Don't use 'N' (now) in the update, fix it to a timestamp that is a multiple of the Interval (300s).

3. Have a 1cdp=1pdp RRA of sufficient size to avoid Consolodation.

For more detail on this, see Alex's famous page here: http://rrdtool.vandenbogaerdt.nl/process.php