Testing Grafana alert returns "Failed to test the rule" with 500 - Internal Server Error response

Question

I'm trying to set up alerts on one of my graphs. I'm using AMG (Amazon Managed Grafana). However, I'm getting "Failed to test the rule" notification. When I inspect HTTP response, it shows

Status Code: 500 Internal Server Error

{"message":"Failed to test rule"}

URL

https://g-39e1d60d36.grafana-workspace.us-east-1.amazonaws.com/api/alerts/test

Here is my alert setup (even If I try something super simple, still getting the same issue):

To me, it seems like Grafana internal error/bug, does anyone experience a similar issue and know the potential resolution?

@TariqN Did this get resolved? We have the same issue on our managed Grafana on AWS. Immediately fails with simple alerts. — Peterdk, Feb 20 '23 at 14:30
Yeah, going back to this question, I manage to resolve it in the same way some time ago. I added a bit more context, but your answer is totally correct. — TariqN, Feb 27 '23 at 17:16

score 1 · Answer 1 · answered Feb 20 '23 at 14:54

1

For us the solution was to not set the datasource dynamically from the environment variable we added in the interface. When we moved the target query to a fixed datasource, it all worked ok.

answered Feb 20 '23 at 14:54

Peterdk

15,625
20
101
140

score 0 · Answer 2 · answered Feb 27 '23 at 16:56

The issue was resolved after I changed the data source of the query from a defined variable e.g ${PrometheusSource} to a specific source instance PrometheusProduction. Seems like Grafana has some issues when setting up an alert for the query where the data source is a predefined variable.

For more context, I'm posting a screenshot from the Grafana dashboard:

After this change, the mentioned issue is not appearing anymore.

Testing Grafana alert returns "Failed to test the rule" with 500 - Internal Server Error response

2 Answers2