I have alert rules in Prometheus and I want to achieve that I don't see the cluster label in the alert but instead I see the env label. I tried to use this code but absolutely no result.
- alert: SQL
expr: avg(rate(pg_stat_database_blks_hit{datname!~"template.*", cluster="prod"}[5m]) / (rate(pg_stat_database_blks_hit{datname!~"template.*", cluster="prod"}[5m]) + rate(pg_stat_database_blks_read{datname!~"template.*", cluster="prod"}[5m]))) by (datname, cluster) < 0.98
for: 2m
labels:
severity: email
env: "{{ $labels.env }}"
annotations:
summary: "PostgreSQL low cache "
description: "PostgreSQL low on cache ..."
relabel_configs:
- source_labels: [cluster]
action: drop
- source_labels: [cluster]
target_label: env
Can anyone advise me how to achieve this?