Stack Overflow
Stack Overflow

Questions tagged [prometheus-alertmanager]

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

Source code is available here: https://github.com/prometheus/alertmanager

753 questions
3
votes
2 answers

Prometheus send resolved notification when metric data missing

We use Prometheus Alertmanager for alerts. Frequently, we are missing metrics because of some connection problems. So, when metrics are missing, Prometheus clear alerts and send resolved alert. After a few minutes, connection problem fixed and…
Melike Sozeri
  • 31
  • 5
3
votes
1 answer

Multiple scrape jobs running on same target in Prometheus

I'm working on use case where I need to scrape metrics at different scrape intervals. For example, metric_one needs to be scraped for every 1 hr and metric_two is scraped for every 15s. Both of these metrics are from the same target. The solution…
Ammar Tahir
  • 33
  • 1
  • 3
3
votes
1 answer

how to exclude some job in count by prometheus query

I have a prometheus query which is as below: 100 * (count by(job, namespace, service) (up == 0) / count by(job, namespace, service) (up)) > 10 and it is giving result {job="kubernetes-pods"} 12.121212121212121 {job="kube-prometheus-kube-proxy",…
user17966463
  • 33
  • 4
3
votes
2 answers

Multiple match clauses in alertmanager config?

In the docs, it says that we can add a set of match clauses under match:. It says literally: A set of equality matchers an alert has to fulfill to match the node. match: [ : , ... ] But it doesn't tell us how to actually…
Sahand
  • 7,980
  • 23
  • 69
  • 137
3
votes
1 answer

Prometheus / Alertmanager: remove link from slack notifications

Is it possible to remove the link that we get when AlertManager posts to Slack? At the moment it's going to http://localhost:9093/.
Snowcrash
  • 80,579
  • 89
  • 266
  • 376
3
votes
2 answers

Configure HTTPS and Basic Auth for Alertmanager

I would like to secure my Alertmanager with tls and authentication, so that in my network, not everyone is able to access the public facing end point. I don't want to use nginx server for proxy. This will be my prometheus configuration: global: …
Supraja Grandhi
  • 71
  • 1
  • 7
3
votes
0 answers

PromQL for K8s pods that are over the CPU / RAM limits

I've deployed kube-prometheus-stack and default rules / alerts helped me a lot, but I need to create custom ones. Specifically, I would like to get information when my pods are 50% (warning) or 70% (critical) over their limits. For example, if…
Rookie
  • 83
  • 1
  • 8
3
votes
4 answers

Prometheus operator AlertmanagerConfig unknown field "webhook_configs"

After applying these manifests https://github.com/prometheus-operator/kube-prometheus/blob/main/kustomization.yaml I want to create AlertManager webhook: apiVersion: monitoring.coreos.com/v1alpha1 kind: AlertmanagerConfig metadata: name:…
Jonas
  • 4,683
  • 4
  • 45
  • 81
3
votes
2 answers

Generate alert if percentage drop is more than 70% for metricX at any point of time for 5 mins

I would like to generate alert for example when counter metricX drops more than 70% at any point of time for 5 mins. Following would be my rule YAML file to generate alert: groups: - name: MetricX dip rules: - alert: MetricX dip by more than…
Santosh M.
  • 2,356
  • 1
  • 17
  • 29
3
votes
1 answer

how to get alerts count ,how many times the alerts fired for last one week in prometheus

sum by(alertname) (changes(ALERTS_FOR_STATE[24h])) this is giving me all the states. i need only like only how many alerts went to critical for last one week (already fired). need assist please. thanks in advance
veera
  • 31
  • 2
3
votes
0 answers

Proper way to configure AlertManager and Prometheus ServiceMonitors with kube-prometheus-stack

I've been reading through the github docs for kube-prometheus-stack trying to figure out the correct way to configure ServiceMonitors and AlertManager. kube-prometheus-stack points back to the prometheus-operator docs for Getting Started and…
Bryan
  • 258
  • 3
  • 11
3
votes
1 answer

Send alert to slack from a Prometheus helm chart

I am trying to create alerts in Prometheus on Kubernetes and sending them to a Slack channel. For this i am using the prometheus-community helm-charts (which already includes the alertmanager). As i want to use my own alerts I have also created an…
Manuel
  • 649
  • 5
  • 22
3
votes
0 answers

DataDog metric query to PromQL

I'm creating alerts in Prometheus and migrating from Datadog. I have two metrics queries that I'm not able to understand yet. avg(last_1d):anomalies(avg:default.burrow_kafka_consumer_lag_total{*} by {consumer_group,env}, 'robust', 3,…
Sam
  • 497
  • 1
  • 10
  • 34
3
votes
1 answer

Alerts firing on Prometheus but not on Alertmanager

I can't seem to find out why Alertmanager is not getting alerts from Prometheus. I would appreciate a swift assistance on this challenge. I'm fairly new with using Prometheus and Alertmanager. I am using a webhook for MsTeams to push the…
Kacey Ezerioha
  • 1,068
  • 4
  • 22
  • 46
3
votes
0 answers

Prometheus AlertManager, firing alert to Slack channel when metrics value changes

I am monitoring a gauge metrics with Prometheus, and I want to fire a alert message to Slack channel every time this metrics value changes, also alert message should contains the current value of metrics. I tried this, here is my rules.yml: -…
cece2048
  • 1,121
  • 2
  • 11
  • 17
1 2 3
…
50 51