Updating rrdtool database

Question

My first post here so I hope I have not been too verbose. I found I was losing datapoints due to only having 10 rows in my rrdtool config and wanted to update from a backup source file with older data. After fixing the rows count the config was created with:

rrdtool create dailySolax.rrd \
--start 1451606400 \
--step 21600 \
 DS:toGrid:GAUGE:172800:0:100000 \
 DS:fromGrid:GAUGE:172800:0:100000 \
 DS:totalEnerg:GAUGE:172800:0:100000 \
 DS:BattNow:GAUGE:1200:0:300 \
 RRA:LAST:0.5:1d:1010 \
 RRA:MAX:0.5:1d:1010 \
 RRA:MAX:0.5:1M:1010

and the update line in python is

newline = ToGrid + ':' + FromGrid + ':' + TotalEnergy + ':' + battNow
UpdateE = 'N:'+ (newline)
print UpdateE
try: 
    rrdtool.update(
    "%s/dailySolax.rrd" % (os.path.dirname(os.path.abspath(__file__))),
    UpdateE)

This all worked fine for inputting the original data (from a crontabbed website scrape) but as I said I lost data and wanted to add back the earlier datapoints. From my backup source I had a plain text file with lines looking like

1509386401:10876.9:3446.22:18489.2:19.0
1509408001:10879.76:3446.99:18495.7:100.0

where the first field is the timestamp. And then used this code to read in the lines for the updates:

with open("rrdRecovery.txt","r") as fp:
    for line in fp:
        print line
## newline = ToGrid + ':' + FromGrid + ':' + TotalEnergy + ':' + battNow
    UpdateE = line
    try: 
        rrdtool.updatev(
        "%s/dailySolax.rrd" % (os.path.dirname(os.path.abspath(__file__))),
        UpdateE)

When it did not work correctly with a copy of the current version of the database I tried again on an empty database created using the same config. In each case the update results only in the timestamp data in the database and no data from the other fields. Python is not complaining and I expected

1509386401:10876.9:3446.22:18489.2:19.0

would update the same as does

N:10876.9:3446.22:18489.2:19.0

The dump shows the lastupdate data for all fields but then this for the rra database

<!-- 2017-10-31 11:00:00 AEDT / 1509408000 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>

Not sure if I have a python issue - more likely a rrdtool understanding problem. Thanks for any pointers.

score 0 · Answer 1 · answered Nov 05 '17 at 23:05

0

The problem you have is that RRDTool timestamps must be increasing. This means that, if you increase the length of your RRAs (back into the past), you cannot put data directly into these points - only add new data onto the end as time increases. Also, when you create a new RRD, the 'last update' time defaults to NOW.

If you have a log of your previous timestamp, then you should be able to add this history, as long as you don't do any 'now' updates before you finish doing so.

First, create the RRD, with a 'start' time earlier than the first historical update.

Then, process all of the historical updates in chronological order, with the appropriate timestamps.

Finally, you can start doing your regular 'now' updates.

I suspect what has happened is that you had your regular cronjob adding in new data before you have run all of your historical data input - or else you created the RRD with a start time after your historical timestamps.

answered Nov 05 '17 at 23:05

Steve Shipway

3,754
3
22
39

Thanks for yr reply. Seems like some of my problems above came from using consecutive RESIZE commands e.g. `1755 rrdtool resize scripts/dailySolax.rrd 0 GROW 1000 1756 rrdtool resize scripts/dailySolax.rrd 1 GROW 1000 1757 rrdtool resize scripts/dailySolax.rrd 2 GROW 1000` which did not grow the number of rows correctly for all RRA's. However now when I create a new RRD and run my python script it does not update the fourth data field. – Rob B Nov 07 '17 at 11:37
When I use : `rrdtool update dailySolaxTemp.rrd 1508371201:10740.95:3430.97:18259.2:58.0` it does the same thing and "58.0" become "NaN" in the rrd. If I have an error in the update line I cannot see it. – Rob B Nov 07 '17 at 11:39
Ok I made an adjustment to the rrd parameters and now I get an error suggesting I have some newline or CR characters causing issues. – Rob B Nov 07 '17 at 12:23
So my line `UpdateE = line` has to be `UpdateE = line.replace("\n", "")` to remove newline character and then the script update works. – Rob B Nov 07 '17 at 12:41

Updating rrdtool database

1 Answers1