Stack Overflow
Stack Overflow

Questions tagged [prometheus-alertmanager]

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

Source code is available here: https://github.com/prometheus/alertmanager

753 questions
0
votes
1 answer

Alerting based on DNS lookup

I currently have some alerting rules based on the "tcp_connect" and "http_2xx" probes that indicate a TCP and HTTP issues on a really high level, not yet to specific alerts. Whenever a probe wasn't able to successfully speak to their target because…
Julian
  • 837
  • 1
  • 11
  • 24
0
votes
1 answer

process-exporter with alert rules in prometheus when process using too much CPU

i am using process-exporter to monitor process, then alert when a process using too much CPU. This is my monitor CPU code in prometheus dashboard sum(rate(namedprocess_namegroup_cpu_seconds_total{groupname=~"$processes",instance="$host",…
Bình Bùi
  • 67
  • 8
0
votes
1 answer

K8S monitoring stack configuration with alerts

I am trying to set up a k8s monitoring stack for my on-premises cluster. What I want to set up is: Prometheus Grafana Kube-state-metrics Alertmanager Loki I can find a lot of resources to do that like: This configures the monitoring stack except…
AnjK
  • 2,887
  • 7
  • 37
  • 64
0
votes
1 answer

Alert manager is not triggering any alerts

There are alerts in Prometheus but slack firing is not happening . Alertmanager says no alerts. i'm attaching the config files of alertmanager and prometheus rules. Need some immediate help as this is a production related…
Sai Prasanth
  • 73
  • 1
  • 13
0
votes
1 answer

Using metric in right hand side of prometheus query

I am using prometheus and grafana to monitor some servers. One of the metrics I have exposed is called recent_tables, which contains the number of assets who have written to sql tables in the past 15 minutes (machines automatically post to sql).…
py_guy_5
  • 41
  • 1
  • 6
0
votes
2 answers

Is there a way to set alert for multiple ENUM metrics with similar name

I'm trying to handle multiple (around 500) metrics with similar names like: INSTANCE03{INSTANCE03="Dead"} == 1 INSTANCE05{INSTANCE05="Dead"} == 1 INSTANCE07{INSTANCE07="Dead"} == 1 Each of them is specified as an Enum which shows status like…
Lordi Wish
  • 3
  • 1
0
votes
1 answer

How to monitor Grafana

I have set up Grafana in my kubernetes cluster for monitoring. But I am really curious about How to monitor the monitoring system(Grafana)?? One solution I can think of is to set up an alert in prometheus-alertmanager so that whenever Grafana pod…
Aman Kr Singh
  • 1
  • 1
0
votes
1 answer

How to change prometheus alert manager port address

I have downloaded prometheus alert manager from prometheus.io and try to run it. Alert manager is not running because some of our internal applications are running on the port 9093. So i need to change the alert manager running port from 9093 to…
sivanesan1
  • 779
  • 4
  • 20
  • 44
0
votes
1 answer

Blackbox Exporter alert with value of the "probe_http_status_code" metric

Currently I have a simple alerting rule set up that uses the "probe_success" metric from Blackbox Exporter to alert when a probe is down, kinda obvious. A requirement is that the status code of the request that failed is included in the alert. So…
Julian
  • 837
  • 1
  • 11
  • 24
0
votes
1 answer

How to fix 'err="yaml: unmarshal errors:\n line 21: field routes not found in type config.plain"'

I'm trying to set up Alertmanager to send to 2 different receivers based on the value of a label in the Prometheus rule. For example, if responsible_team label equals web, send to the web team receiver. If reponsible_team label equals database, send…
mr_x
  • 1
  • 1
  • 1
  • 1
0
votes
2 answers

Is it possible to avoid sending repeated Slack notifications for already fired alert?

Disclaimer: First time I use Prometheus. I am trying to send a Slack notification every time a Job ends successfully. To achieve this, I installed kube-state-metrics, Prometheus and AlertManager. Then I created the following rule: rules: - alert:…
ThatChrisGuy
  • 559
  • 2
  • 7
  • 23
0
votes
1 answer

Muti DC Alertmanager HA setup

I have few DC clusters, each one have own set of Prometheus-es From what I've read here https://www.robustperception.io/prometheus-and-alertmanager-architecture it is important to have one alertmanager cluster to prevent alerting from each DC. But…
sev3ryn
  • 1,055
  • 1
  • 9
  • 17
0
votes
1 answer

Connecting prometheus-alertmanager with zoom via an incoming-webhook

Hello I'm trying to start sending alerts to Zoom using prometheus-alertmanager's webhooks, but I keep getting error msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected…
Yordan Yordanov
  • 201
  • 1
  • 2
  • 6
0
votes
1 answer

How to make Prometheus send Alerts to two different Alertmanagers based on Alert labels?

I have the following two alerts: alert: InstanceDown expr: (up == 0) and (team == "foo") for: 5m labels: severity: page team: foo annotations: summary: "Instance {{$labels.instance}} down" alert: InstanceDown expr: (up == 0) and (team ==…
Maklaus
  • 538
  • 2
  • 16
  • 37
0
votes
1 answer

Multiple query in prometheus throwing No datapoints found error

Facing error while running two query at once. Query1: sum by(container_label_io_kubernetes_pod_name) (rate…
Prakash Kumar
  • 2,554
  • 2
  • 18
  • 28
1 2 3
…
50 51