2

I am using Grafana 9.4 for dashboarding and alerting. I currently have an alert set up with:

  • Query A:
    • a Prometheus metric with labels like {tenant="tenant1", ...} and legend set to {{tenant}}.
  • Expression B (the alert condition):
    • Classic_conditions WHEN last() of A is above 100

When triggering, the alert currently generates a notification similar to:

Firing
Value: B0=222.28333333333333
Labels:
- alertname = My alert name
- grafana_folder = My folder
- team = my-team
Annotations:
- summary = My summary
...
  1. Would it be possible to have a dynamic label that would contain my tenant's name? That way, I could silence the alert for a given tenant only? I tried a few values for the label thinking that some sort of templating was used but I couldn't get the proper syntax to have a value.
  2. How can I at least replace the B0 in the notification with the actual tenant name from query A?
Marcel Gosselin
  • 4,610
  • 2
  • 31
  • 54

1 Answers1

3

You have to use Grafana unified alerting (not old Grafana legacy alerting).

  1. A: use query, which will group by tenant label
  2. B: use expression with Operation: Reduce, Function: Last
  3. C: use expression with Operation: Math, Expression: $B>100

Set alert condition: C - expression.

Use Preview alerts to verify that all labels are generated correctly.

Tenant label will be available in the alert labels section. Of course you can use it as {{ $labels.B.tenant}} in the alert Description or Summary field.

Jan Garaj
  • 25,598
  • 3
  • 38
  • 59
  • How can I tell whether I am using unified or legacy alerting? Your solution fixed the point 1 of my question in the sense that I now have a label `tenant=something`. The `{{ $labels.B.tenant}}` in the summary yielded `[no value]` but since I got a new label named `tenant` I tried `{{ $labels.tenant }}` and it worked! – Marcel Gosselin May 17 '23 at 13:25
  • @MarcelGosselin use `Preview alerts` and verify where are labels generated. – Jan Garaj May 17 '23 at 14:26