RRD database empty although updating

Question

I am trying to fill a rrd database based on duration and intervals the user commits through a gui. i built a timer which calls the update-function every ... minutes. After the measurement I try to look at the collected data by using the fetch-method but I get only "none"-values listed.

def update(self):
    t=getTemperature()
    h=getHumidity()
    if self.status==11:
        rrdtool.update(self.path,"N:"+t+":"+h)
    elif self.status==10:
        rrdtool.update(self.path,"N:"+t)
    elif self.status==01:
        rrdtool.update(self.path,"N:"+h)

The sensor values are correct, I checked it via a print-out. The rrdtool.create-method works also totally fine, a database is created, but as i said not actually updated correctly.

Here the usage of rrdtool.create():

def __init__(self,term,temEnabled,humEnabled,mins,hrs,fre):
    self.path="/home/pi/Wetterstation/Speicherort/"+term+".rrd"
    if os.path.exists(self.path)==False:
        totalEntries=int((mins+hrs*60)/fre)
        totalEntries=str(totalEntries)
        if temEnabled==True and humEnabled==True:
            rrdtool.create(self.path,
                           "DS:temperature:GAUGE:900:0:50",
                           "DS:humidity:GAUGE:"+str(fre)+":0:100",
                           "RRA:AVERAGE:0.5:1:"+totalEntries,                                       "RRA:MIN:0.5:12:2400",
                           "RRA:MAX:0.5:"+totalEntries+":1",
                           "RRA:MIN:0.5:"+totalEntries+":1")
            self.status=11
        elif temEnabled==True:
            rrdtool.create(self.path,
                           "DS:temperature:GAUGE:"+str(fre)+":0:50",
                           "RRA:AVERAGE:0.5:1:"+totalEntries,
                           "RRA:MIN:0.5:"+totalEntries+":1",
                           "RRA:MAX:0.5:"+totalEntries+":1")
            self.status=10
        elif humEnabled==True:
            rrdtool.create(self.path,
                           "DS:hum:GAUGE:"+str(fre)+":0:100",
                           "RRA:AVERAGE:0.5:1:"+totalEntries,
                           "RRA:MIN:0.5:"+totalEntries+":1",
                           "RRA:MAX:0.5:"+totalEntries+":1")
            self.status=01
        self.success=True
    else:
        self.success=False

What is the value of 'fre'? This is your humidity heartbeat. What is totalEntries? This is the size of your RRAs. Finally, how often are you calling update(), and how many samples are you storing before you try to view the data? Possibly you have not been submitting data for long enough (45 min or more) to have RRA entries? Can you show the list of stored data, and the command you used to extract - possibly you are querying a different time window to the stored data? — Steve Shipway, Jun 19 '17 at 00:57
Also, you seem to hardcode a heartbeat of 900 if both DS are present, but use 'fre' otherwise. This might be causing you issues if you are getting a different heartbeat to what you expect. You don't specify a step, so are probably getting 300 as the default, but your RRA calculations are based on the heartbeat when they should be on the step... — Steve Shipway, Jun 19 '17 at 01:18
@SteveShipway "fre" is the frequency submitted in minutes, "totalEntries" calculates the amount of times I save a value. I call the method update every -fre- minutes. I submit data only for a short time (while testing), e.g. one minute. The 900 I forgot to replace with "fre", as step I define now fre too. — byteshark17, Jun 20 '17 at 09:38
@SteveShipway I extracted the data with the fetch-method - without specifying any time. I added a link to the picture below. http://de.tinypic.com/r/rcot4n/9 — byteshark17, Jun 20 '17 at 09:55
@SteveShipway is there a minimum time I have to submit data in the RRD? The weather station should be able to work for short time as for long time measurements. If yes, is there any alternative option to save data in a database and later on create graphs locally (as a png or jpg)? — byteshark17, Jun 20 '17 at 18:14
I've put more detail in the answer, but basically you must submit the data with no longer an interval than the heartbeat. Data should come at approximately an interval of the step size, as regularly as possible. You should also make sure that when extracting data, you specify the same time window as you used when submitting it! — Steve Shipway, Jun 21 '17 at 21:58

score 0 · Answer 1 · answered Jun 21 '17 at 21:14

Your problem is a misunderstanding of the relationship between the step size, DS heartbeat, and RRA XFF setting. Although you don't tell me what your 'fre' value actually is, it is clear that the RRD definition is not correct.

The step size is the smallest time interval in the RRD file. By default, this is 300sec (5min) though you can specify it smaller or larger if necessary at creation time. Generally, you would set this to be the expected amount of time between data samples; so if your samples come approximately every 5min, you set this to 5min.

The heartbeat is the maximum amount of time between samples before they are deemed unknown. This is an indicator of the irregularity of samples. In general, this would be twice the expected sample interval; so if your samples come every 5min, set this to 600s (10min). If there is a gap larger than this, the sample range is marked 'unknown'.

The XFF is the proportion of DS data points making up an RRA consolodated data point which can be unknown before the consolodated data point becomes unknown. Generally, this is set to 0.5, but can be higher if you want (e.g. 0.9). Your RRA used a consolodation factor of 1, so in this case, the XFF is irrelevant, but it is more important when you start to summarise.

Since you have your heartbeat set to fre, which is the approximate frequency of arriving data samples, it is unlikely that you will have enough samples at a small enough interval for there to be any stored as anything other than 'unknown'. Remember that RRD will normalise and consolodate the data, and convert internally to a rate, when data arrive. This means that the incoming data need to be fairly regular. If sample intervals have a high standard deviation then you will need a high heartbeat and you will get a lot of adjustments made.

In your case, I would suggest you set the RRD step size to either 60 or 300 (whichever is less than fre), and the heartbeat to 2*fre.

RRD database empty although updating

1 Answers1