2

I’ve a request to include the time when an alert was raised in the message body of the email for that alert.

I’ve checked out Create Alerts and Alert notification templating and it looks like it might be possible, but I can’t find any reference to any list of supported variables.

For reference, we're using Grafana v7.4.0 (c2203b9859) and creating an alarm off a cloudwatch metric query:

REMOVE_EMPTY(SEARCH('{"AWS/ApiGateway","ApiName","Method","Resource","Stage"} MetricName="5XXError"', 'Sum', 60))

So there's nothing in the results we can use for this, unless anyone knows of a way to pull the last timestamp out of one of the datatables ;)

What I’m trying to do is include something like this, in the email being sent out when this alert is triggered (although the actual date format doesn’t matter at all):

Alert raised at 2021-08-26T12:03

Thanks

Daniel Bray
  • 21
  • 2

1 Answers1

0

You can do this with custom email templating in grafana. The grafana uses 'go templating' for this.

To add custom template do this: go to alerting -> contact points -> add template -> give a template name (say email.body), write content of template in 'go templating' format (Read further to know what to write in content). Further you may also create another template for subject line of email, say email.subject. (https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/)

To use this templates in email, Contact points -> edit contact point -> Optional email settings -> In message field add {{template "email.body" .}} and in subject field add {{template "email.subject" .}}

As content for email.subject template, you can use following from example in grafana documentation:

{{ define "email.subject" }}
  {{ if .Alerts.Firing -}}
    {{ len .Alerts.Firing }} firing alert(s)
  {{end}}
  {{ if .Alerts.Resolved -}}
    {{ len .Alerts.Resolved }} resolved alert(s)
  {{end}}
{{ end }}

Coming to the question, there are two cases of timestamps:

1. If you want the time at which the alert was activated in grafana:

As an example add this to content of email.body template

THE TIME OF EVENT IS: {{.Alerts.StartsAt}}

2. If you want to add some time from the log string:

This functionality is very limited. So this is only a work around. We can extract some information from the {{.Alerts.ValueString}}. The value in this variable depends on what query you configured in grafana alert. I had some logs as result in alert query (from elasticsearch data source). So I was getting the logs as simple strings in this variable. So I tried to extract time from it using slice function of go templating.

As an example suppose the value of .Alerts.ValueString is "Count 2023-05-10 14:59:22,166 ERROR [main]...", then {{ slice .Alerts.ValueString 6 25}} will output "2023-05-10 14:59:22".

Add following as content of email.body template:

THE TIME OF EVENT IN LOG IS: {{ slice .Alerts.ValueString  6 25}}
THE ValueString IS: {{.Alerts.ValueString}}

As per my current knowledge, it is only possible to use default available functions of go templating and using a custom function seems difficult or no documentation available. If using custom go functions in grafana template was possible, everything would have been easier, using regex.

SFA
  • 63
  • 5