Stack Overflow
Stack Overflow

Questions tagged [prometheus-alertmanager]

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

Source code is available here: https://github.com/prometheus/alertmanager

753 questions
5
votes
0 answers

Line break in Prometheus' HTML Alert / Rule view

I have several Prometheus alerting rules which are quite long. In my editor I manage to add line breaks to the rules' expressions (see line expr) with a | at the beginning: - alert: IF_Critical annotations: description: The service on {{…
kingisch
  • 71
  • 4
5
votes
2 answers

Adding Alertmanager expression queries while alerting

My intention is to add the query expressions to be displayed in descriptions or summary when alertmanager is alerting through alerting mediums. Is it possible something like this below? - alert: OutOfDiskSpace expr: node_filesystem_free_bytes /…
td4u
  • 402
  • 5
  • 17
5
votes
1 answer

Is it possible to conditionally alert with prometheus based on if another alert is firing or not?

The idea here is that I'm considering alerting on a metric that corroborates a metric that another tool is monitoring. For example: I might have a service doing some task by feeding off a queue and when processed tasks drop to zero an alert needs…
lucidquiet
  • 6,124
  • 7
  • 51
  • 88
5
votes
2 answers

CPU Load average rule for 5 minutes

We are using Prometheus-Grafana. Now we want to set alert for CPU load average of 5 minutes. We have 60 servers which have different CPU core like few machine have 1 core, 2 core, 6 core, 8 core etc. The below Rule will give the result for load 5…
Tekchand Dagar
  • 317
  • 1
  • 7
  • 18
5
votes
1 answer

Is the fingerprint field in Alertmanager unique?

I am developing a dashboard, which receives all Alertmanager readings and processes them. I looked for a unique field in the request payload to create a unique external alert id in my database. The request payload looks something like this: { …
mhyousefi
  • 1,064
  • 2
  • 16
  • 30
5
votes
1 answer

How to display values of metrics in alert rules prometheus

I create a rule in prometheus alertmanager, which tell about least space on mountpoints in percents - and additional to this i want to show how much least space in gigabytes , but i do not want to hardcoded a mountpoint to show gigabytes, i want use…
a1dude
  • 174
  • 2
  • 13
5
votes
1 answer

How to inhibit alerts outside business hours with prometheus alertmanager?

Our application relies on data sources that are only active during business hours. We have alerts setup in Prometheus to notify us when streams dry up. However, we don't want to get "false" alerts outside business hours. I followed this post to…
Jeroen Rosenberg
  • 4,552
  • 3
  • 27
  • 39
5
votes
2 answers

PromQL. How to add label value from different metric

If I have two metrics: kube_pod_container_status_restarts_total {..., pod="my_pod_name_42", ...} and container_memory_usage_bytes {..., pod_name="my_pod_name_42", cluster_name="megatron", ...} And I want to set up alert for restart metrics, but…
Andrii Rzhepishevskiy
  • 51
  • 1
  • 1
  • 3
5
votes
3 answers

Prometheus to get trigger an alert when node is in unschedulable state

Can someone help me in writing a promql query to get the node status whether node is schedulable or not. Thanks in advance.
vijay kumar
  • 61
  • 2
  • 5
5
votes
1 answer

Complex rules/filters for Prometheus-Alertmanager Alerts

Situation: I have Prometheus and Alertmanager setup to monitor, among other things, CPU temp of various devices. Alertmanager sends alerts from production devices to PagerDuty. The devices I'm monitoring have different models with different…
Max
  • 57
  • 1
  • 9
4
votes
2 answers

How to get all metric names from Prometheus server filtered by a particular label

We want to get all metric names from Prometheus server filtered by a particular label. Step 1 : Used following query to get all metric names, query succeeded with all metric names. curl -g 'http://localhost:9090/api/v1/label/__name__/values Step 2…
user3082802
  • 47
  • 1
  • 1
  • 6
4
votes
1 answer

Time based alerts in prometheus alertmanager

I want to set an alert in Prometheus alertmanager that has to be triggered between a given time Eg: Condition to be checked between 09:15 to 15:30
ARK
  • 57
  • 1
  • 3
4
votes
3 answers

Use conditional operator in Prometheus alert rules to set severity

I would like to use the conditional operator into the Prometheus alert.rules definition to set a specific severity. For instance, if the environment is production, I want to set the severity to critical else another value. Something like: - alert:…
rschirin
  • 1,939
  • 10
  • 34
  • 44
4
votes
2 answers

Prometheus alertmanager persistent storage with docker

For some reason I can't get persistent storage to work with alertmanager. This is my compose: alertmanager: image: 'my/alertmanager/prod:latest' restart: always volumes: - alertmanager-data:/alertmanager-data command: -…
somerandomusername
  • 1,993
  • 4
  • 23
  • 55
4
votes
1 answer

AlertManager is not forwarding alerts to webhook receiver

We have setup Prometheus in a kubernetes cluster using PrometheusOperator. We are trying to configure AlertManager using the AlertManagerConfig custom resource. We tried creating an alert route which maps to a webhook receiver and then triggering a…
aaruja
  • 371
  • 2
  • 12
1 2
3
…
50 51