I have looked at this question/answer here: Prometheus AlertManager - Send Alerts to different clients based on routes
And it was a pretty good start for me, and I wish I could've commented a quick question to the Answerer there but I don't have the rep.
Anyways, I have an alert.rules.yml file with two groups which looks like:
groups:
- name: DevOpsAlerts
rules:
- alert: InstanceDown
expr: up == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} down"
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes. ({{ $value }} minutes)"
- alert: InstanceHighCpu
expr: 100 - (avg by (host) (irate(node_cpu{mode="idle"}[5m])) * 100) > 5
for: 10m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.host }}: CPU High"
description: "{{ $labels.host }} has high CPU activity"
- name: TestTeam2
rules:
- alert: - alert: InstanceLowMemory
expr: node_memory_MemAvailable < 268435456
for: 10m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.host }}: memory low"
description: "{{ $labels.host }} has less than 256M memory available"
- alert: InstanceLowDisk
expr: node_filesystem_avail{mountpoint="/"} < 1073741824
for: 10m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.host }}: low disk space"
description: "{{ $labels.host }} has less than 1G FS space"
Along with that I have an alertmanager.yml file which looks like
global:
smtp_smarthost: 'smtpserver'
smtp_from: 'alertsender@email.com'
smtp_auth_username: 'alertsender@email.com'
smtp_auth_password: 'verystrongpassword'
smtp_require_tls: maybe
route:
group_by: ['alertname', 'cluster', 'service']
#default receiver
receiver: DevOps
routes:
- match:
alertname: InstanceDown
receiver: DevOps
- match:
group: InstanceHighCpu
receiver: test-team-1
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'cluster', 'service']
receivers:
- name: DevOps
email_configs:
# - to: devops_dude@email.com
- name: test-team-1
email_configs:
- to: test-dude1@email.com #This can be any email specified from the team
- name: team-name-2
email_configs:
- to: test_email@test.com #This can be any email specified from the team
So from what I've gathered I am able to route alerts to specific receiver groups by specifying an alert name from the alert rules file and routing that to a specific receiver.
The big question I really have is: Is there a way to route alerts to specific receivers based on group names as opposed to alert names from the alert rules file.
So instead of
routes:
- match:
alertname: InstanceDown
receiver: DevOps
Is there some sort of way to implement:
routes:
- match:
group: DevOpsAlerts
receiver: DevOps
I've been scouring the internet for some sort of example like this but I couldn't find anything. Thanks.
