1

I'm adding an interval variable to a dashboard to allow averaging some prometheus time series over an interval to get cleaner looking graphs. But I want to leave in the base sample rate (15s) so the graph can be viewed without averaging.

However, some graphs use rate() which gives no data for the base sample rate.

How can I share $interval for all time series in the dashboard, but force graphs using rate() to use a minimum interval (e.g., 1m) ? Is there something I can add to the query like max($interval,1m)

Just to add a little more detail...the data for the time series for one of the graphs looks like: 100 0 100 0 100 0 ... and I want to average that to 50 (they are not all that clean) by using a time interval larger than $__interval.

I suppose what I need to do is use a grafana interval variable (call it $myinterval) which I already have and make a chain variable $myinterval_rate but I'm not sure how to calculate it in grafana.

user10489
  • 410
  • 1
  • 3
  • 12
  • AFAIK, you can't use functions inside of range selector. And why do you think that `$__interval` and `$__rate_interval` are not suitable for you? – markalex Jul 30 '23 at 05:47
  • I was hoping I could do something in grafana rather than promql, I know it has to be a constant for promql. I don't want to use those two variables because they are used to adjust the graphs to match the screen resolution -- I want to adjust it for other reasons. (Actually, it would be ok to use those as _default_ values.) – user10489 Jul 30 '23 at 13:09
  • "by using a time interval larger than $__interval" `$__rate_interval` [is larger than `$__interval`](https://grafana.com/docs/grafana/latest/datasources/prometheus/template-variables/#use-__rate_interval). I still believe you most probably can get by with built-in variables (and maybe tinkering with query options, like min step). – markalex Jul 30 '23 at 15:00
  • And regarding variable chaining: it is not trivial, but [can be done] with some level of ingenuity. Remember, that technically time range in Prometheus are strings, and not numbers, so you might need quite a bit of operations to emulate something like `max`. Or you can limit input and output to some set of predefined values. – markalex Jul 30 '23 at 15:03
  • Using rate_interval is not helpful, because 1) the period of the fluctuations is variable and I want to be able to user select the averaging interval and 2) the graph needing the variable interval most is not a rate graph (and I want the rest of the graphs to align with it, hence trying to use the parameter in all graphs...) – user10489 Jul 30 '23 at 16:46
  • One more thing: will it be sufficient for you to have list of predefined pairs, like `[(1m, 2m), (2m, 4m), (3m, 5m)]`, where one is like `$myinterval` and another `$myinterval_rate`? – markalex Jul 30 '23 at 16:58
  • Because if yes, you can define [key-value variable](https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/#add-a-custom-variable), and with [this approach](https://community.grafana.com/t/referencing-key-value-variable-with-another-variable-as-the-key/82362/2) use both keys and values in your queries. – markalex Jul 30 '23 at 17:07
  • Interesting...that might work. I see grafana has provisions for key-value pairs, but I don't see documentation that a tuple can be used for the value, and I don't see a way to extract a member out of a tuple. An answer would have to show how to do this... – user10489 Aug 03 '23 at 11:52
  • I bet you can use your typical query with `$__rate_interval` just make sure you set "min interval" in query options to 2-3 times the scrape interval. Query options are per-panel settings, it's near the datasource drop-down and above the field where you write the query. – anemyte Aug 04 '23 at 06:13
  • $__rate_interval doesn't do what I want, and has nothing to do with this question. Also, min interval probably isn't appropriate either because the panels have multiple data sources, some of which are rates and some of which are averages of gauges. – user10489 Aug 04 '23 at 22:44

1 Answers1

1

You can create custom variable with key : value pairs, and use its "keys" for "averaging" functions, and "values" for rate-related functions.

This can be done in the following manner:

  1. Create custom variable with following values:
15s : 1m, 30s : 1m15s, 1m : 2m, 2m : 2m15s

Notice spaces surrounding colon, they are important for values parsing. Also, notice, that values of different pairs should be different.

  1. Go to your queries:

2.1. For those that use you averaging use variable in the form ${myvar:text}, where myvar is the name of created variable. This will substitute "key" of the variable. For first pair in my example: 15s.

2.2. For queries that use rate-like functions use variable in the form ${myvar:value}. This will substitute "value" of the variable. For first pair in my example: 1m.

If uniqueness of values for the variable is a deal breaker for you, you can use chaining of the variables.

For this case query will look something like this:

query_result(
  (absent(non_existent{pseudo_label="1m"}) and on() (absent(non_existent{pseudo_input="15s"}) and absent(non_existent{pseudo_input="$myvar"}))) or
  (absent(non_existent{pseudo_label="1m"}) and on() (absent(non_existent{pseudo_input="30s"}) and absent(non_existent{pseudo_input="$myvar"}))) or
  (absent(non_existent{pseudo_label="1m"}) and on() (absent(non_existent{pseudo_input="1m"}) and absent(non_existent{pseudo_input="$myvar"}))) or
  (absent(non_existent{pseudo_label="2m"}) and on() (absent(non_existent{pseudo_input="2m"}) and absent(non_existent{pseudo_input="$myvar"})))

where $myvar is a variable with your initial intervals.

markalex
  • 8,623
  • 2
  • 7
  • 32
  • 1
    Note: didn't work first try. It's important to change variable type from interval to custom. This also loses the auto option (which isn't great anyway). – user10489 Aug 04 '23 at 04:40