Stack Overflow
Stack Overflow

Questions tagged [prometheus-alertmanager]

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

Source code is available here: https://github.com/prometheus/alertmanager

753 questions
4
votes
2 answers

Loki not alerting Alertmanager

I am new with Loki and have made an alert in Loki but I don't see any notification in the Alertmanager. Loki is working fine (collecting logs), Alertmanager also (getting alerts from other sources), but the logs from loki don't get pushed to…
OmarLittle
  • 423
  • 1
  • 9
  • 18
4
votes
1 answer

sum of rate function in prometheus

Given the following prometheus time series called requests : the vector query requests[3 seconds] is : and the rate of the vector query requests[3 seconds], rate(requests[3 sec]) (computed by the shown formula) is : My question is : what is…
Mazen Ezzeddine
  • 662
  • 1
  • 8
  • 28
4
votes
4 answers

How to fix the error undefined variable "$labels"in Prometheus?

I used this alret - alert: my alert expr: status{status="ERROR"} for: 30m labels: severity: WARNING annotations: myData: "{{ $labels.myData }}" myData2: "{{…
user1365697
  • 5,819
  • 15
  • 60
  • 96
4
votes
0 answers

How to get Custom Metrics from Azure app-insight in Prometheus server?

Because set alert rule in Azure is too expensive, I want set alert rule in Prometheus instead. However, is there any way to expose the metrics on Azure(app insight) to Prometheus server? (I already have some application running on Azure now and have…
Ricky Parker
  • 133
  • 2
  • 8
4
votes
2 answers

Monitor the status of batch jobs using metrics in Prometheus

Several of our applications have batch jobs that aggregate data every night. These batch jobs, which are Python scripts, use Prometheus Pushgateway to push metric values to Prometheus, and we have rules that trigger alerts (in Alertmanager) when…
daniel238
  • 43
  • 1
  • 3
4
votes
1 answer

Delay Prometheus alert before changing from active to inactive

I have an alert in my Prometheus set up that sends an alert when someMetric > 100 has been valid for 5m and then resends the alert every 24h according to the configuration below: prometheus-alert.yml - alert: TestAlert expr: someMetric >…
Cleared
  • 2,490
  • 18
  • 35
4
votes
2 answers

How to config email alert in using grafana and prometheus-operator

I installed prometheus-operator (include prometheus/alertmanager/grafana) via helm. Then I access Grafana UI and config alert via email. When I click send an email test, I got the message “ SMTP not configured, check your grafana.ini config file’s…
taibc
  • 897
  • 2
  • 15
  • 39
4
votes
2 answers

Error parsing alert "expr" with regular expression in it

I'm trying to define an alert rule with "expr" containing regular-expression Metric: XYZ-POST-failure-400-1min-rate - alert: alert_name_here expr: __name__=~"(.*)-POST-failure-\d{3}-1min-rate" > 0 Unit testing this using "promtool" Error group…
user2922123
  • 51
  • 1
  • 2
4
votes
3 answers

Test Rules AlertManager FAILED: yaml: unmarshal errors: line 1: field groups not found in type main.unitTestFile

please help i got error message when testing alert manager below promtool check rules /etc/prometheus/alert.rules.yml Checking /etc/prometheus/alert.rules.yml SUCCESS: 3 rules found promtool test rules /etc/prometheus/alert.rules.yml Unit…
Fajar Hadiyanto
  • 81
  • 2
  • 4
4
votes
1 answer

route matching multiple labels

I am not getting alertmanager to send alerts based on multiple labels. In general sending e-mails on alerts is working. But only if there is a simple match on one label. E.g. teamB route is working. teamA route is not. route: receiver:…
Christian Schyma
  • 190
  • 1
  • 2
  • 16
4
votes
0 answers

Context deadline exceeded for Prometheus service endpoint

Prometheus Version - 6.7.0 In order to scrape only specific port for a service, I have used below configuration. In Prometheus the target service endpoint always goes in status context deadline exceeded and Error says - Get…
Darshan Deshmukh
  • 353
  • 1
  • 4
  • 15
4
votes
0 answers

Prometheus check if a metric disappear

I want have make alerts based on a metric which in my case, sometimes, may disappear Name of metrics count_items returns number every 10minutes if number < 750000 an alarm is triggered if a metric not exists = OK or metrics count > 750000 = OK I…
Djam36
  • 53
  • 6
4
votes
2 answers

Prometheus AlertManager - Send Alerts to different clients based on routes

I have 2 services A and B which I want to monitor. Also I have 2 different notification channels X and Y in the form of receivers in the AlertManager config file. I want to send to notify X if service A goes down and want to notify Y if service B…
Janshair Khan
  • 2,577
  • 4
  • 20
  • 44
3
votes
0 answers

Configure Loki log-based alerts with kubernetes CRD

I like the idea of Prometheus Operator where alerting configurations are deployable kubernetes objects defined with CRDs. apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: example role: alert-rules …
pizycki
  • 1,249
  • 4
  • 14
  • 26
3
votes
1 answer

Alert based on metric value compared to one of its own label values

I am using kube-prometheus-stack and the yaml snippets you see below are part of a PrometheusRule definition. This is a completely hypothetical scenario, the simplest one I could think of that illustrates my point. Given this kind of…
igg
  • 2,172
  • 3
  • 10
  • 33
1 2 3
…
50 51