How do I use win.disk.duration in Bosun or how do I downsample a counter type metric?

Question

I'm using Bosun to gather information about the average response time of a hard disk (win.disk.duration), it produces a graph such as:

I understand that this is showing the rate of change of the WMI reported value, AvgDiskSecPerRead.

What I want to do is downsample this calculated value taking the max over a time period. However if I set a downsample of Max with a window of 5m I get this:

Note the Y-axis scale change.

How can I downsample the rate to get the results I'm expecting (or why doesn't what i'm asking for make sense)?

Answer 1

Currently since this is a counter you can't downsample it correctly with OpenTSDB (the datasource I assume you are using due to the question tags). This is because OpenTSDB got the math wrong during design and that hasn't been fixed yet. More information in this Google groups thread. In short, the rate calculation (derivative) needs to be earlier in the order of operations.

Also, be aware that currently on our side (scollector) we seem to have messed up a related metric win.disk.percent_time which still has to be looked into (tracked in this issue).

Answer 2

It seems that this transformation is about applying aggregation to the first derivative. Here's an example using win.disk.duration metric from scollector but on a different backend.

1. Raw data represents a counter with reset.
2. First derivative calculates the rate of change over a user-defined interval (1 second). Negative delta is handled as a reset by default, but this behavior can be controlled with rate-counter=true|false parameter.
3. The aggregation function max is applied to the rate over a user-defined period (5 minutes).

I would recommend using 95 percentile function instead of max however. The p95 statistic allows you to discards infrequent spikes from analysis.

Disclaimer: I work for Axibase.