Stack Overflow
Stack Overflow

Questions tagged [prometheus-alertmanager]

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

Source code is available here: https://github.com/prometheus/alertmanager

753 questions
2
votes
1 answer

Alertmanager not sending out alert to slack

I configured alertmanager with prometheus and according to the prometheus interface the alerts are firing. However there is not slack message showing up and I am wondering if maybe ufw needs to be configured or if there is any other config I…
merlin
  • 2,717
  • 3
  • 29
  • 59
2
votes
1 answer

Excluding a label value from a specific host only

I've below metrics in Prometheus {hostname="HOST-A",mountpoint="/DIR/1"} 0 {hostname="HOST-B",mountpoint="/DIR/2"} 0 {hostname="HOST-B",mountpoint="/DIR/7"} 0 {hostname="HOST-C",mountpoint="/DIR/3"} 0 {hostname="HOST-D",mountpoint="/DIR/4"} 0 How…
Avin Varghese
  • 4,340
  • 1
  • 22
  • 31
2
votes
0 answers

Expression in Prometheus to determine constant growth over time

We get metrics about the number of messages in queues from our instance of ActiveMQ Artemis 2.10.0 to Prometheus, and I need to be notified when for a certain amount of time (let's say 8 hours) the queue grows and does not decrease (usually this…
Skulfinder
  • 23
  • 2
2
votes
0 answers

How to get the full log error message from Grafana into Alertmanager for alerting?

I can get the labels from the log messages on Grafana for which I am alerting through alertmanager. However, I want to get the full log message into the alert message that gets sent to slack or email. Is it possible somehow? I have tried…
John
  • 21
  • 1
2
votes
0 answers

How to silence Alertmanager specific rule?

I have an alert concerning the increase in load of the CPU of my servers. Every day, one of them, a jenkins, with the builds, causes alerts. I would like to mute them, because I know full well that during these builds the load is high. However, I do…
erwann le guevedec
  • 31
  • 1
  • 3
2
votes
1 answer

Prometheus Alertmanager is always sending notifications

Is there any way for alertmanager to stop firing alerts continuously once one has been triggered? For example I have set some alerting rules for checking uptime of services. If one of those services is down, then alertmanager will not stop sending…
ageladas
  • 144
  • 1
  • 10
2
votes
4 answers

How to run pod based on Prometheus alert

Is there any way we can run pod based on the alert fired from Prometheus? We have a scenario where we need to execute a pod based on the disk pressure threshold. I am able to create alert but I need to execute a pod. How can I achieve that? groups: …
Rajendar Talatam
  • 250
  • 1
  • 12
2
votes
1 answer

Prometeus Failed to start TCP listener on \"0.0.0.0\" port 9094:

When I'm trying to run Prometheus alert manager with alertmanager --config.file /etc/alertmanager/alertmanager.yml It giving me error like $ alertmanager --config.file /etc/alertmanager/alertmanager.yml level=info ts=2021-11-10T05:08:13.937Z…
Hack-Z
  • 47
  • 7
2
votes
1 answer

Why does Prometheus resolve unresolved alerts?

I have job failure alerts in prometheus, which resolves itself right after 2 hours I got the alert where the alert actually is not resolved. How come Prometheus resolves it? Just so you know, this is only happening with this job alert. Job Alert: …
cosmos-1905-14
  • 783
  • 2
  • 12
  • 23
2
votes
1 answer

alertmanager filter by tag (timescale backend)

I am using alertmanager configured to read from a timescale db shared with other Prometheus/alertmanager systems. I would like to set/check alerts only for services including a specific tag, therefore wondering how could I configure prometheus to…
nbari
  • 25,603
  • 10
  • 76
  • 131
2
votes
0 answers

Prometheus custom Alert annotations based on the pod

So we have a K8s cluster, where we run Pods with sort of the following spec: apiVersion: v1 kind: Pod metadata: annotations: prometheus.io/path: /metrics prometheus.io/port: "9115" prometheus.io/scrape: "true" …
chepeftw
  • 21
  • 3
2
votes
1 answer

How to match more than one labels in Prometheus alerts?

I want to send notifications either it has Level1 or Level2. But I'm not sure if Prometheus tries to find two of the labels at the same time, which is not the case. How can I do that? - receiver: 'alert' match: severity: Level1 severity:…
cosmos-1905-14
  • 783
  • 2
  • 12
  • 23
2
votes
0 answers

Prometheus Alertmanager Cluster mode sending duplicate alerts

I have a monitoring system using 2 EC2 instances, each running prom/prometheus:v2.30.0 and prom/alertmanager:v0.23.0 and both of the prometheus instances are scraping the same metrics sending the alerts to both alertmanagers: # Alertmanager…
Hugo Dias
  • 341
  • 3
  • 13
2
votes
1 answer

AlertManager - Send alerts to different receivers based on routes for particular Jobnames

I have configured prometheus alertmanager on Ubuntu server to monitor multiple azure vms. Currently all the vm instance alerts are notified to a default email group. I need to trigger alert to Team A(user1,user2,user3) & default group if Server A…
Pratik M
  • 70
  • 2
  • 9
2
votes
1 answer

AlertManager notification from cAdvisor container

I am using the common monitoring tools (Prometheus, cAdvisor, AlertManager), and I faced this issue that one of the servers firing each 30min containerCpuUsage but unfortunately I do not know which container is this (I am guessing this is the…
MK83
  • 1,652
  • 1
  • 11
  • 11
1 2 3
…
50 51