2

although I can see my alarms on prometheus, I cannot view alarms via alertmanager, although their status is Firing. My settings;

enter image description here

enter image description here

prometheus.yml config

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093

rule_files:
  - alert.rules.yml

scrape:
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'alertmanager'
    static_configs:
    - targets: ['localhost:9093']

alertmanager.yml config

global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.com'
  smtp_require_tls: false
  
  slack_api_url: 'https://hooks.slack.com/sxx'

route:
  group_by: ['instance', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: team-1
  
receivers:
  - name: 'team-1'
    email_configs:
      - to: 'your-email-address'
    slack_configs:
      - channel: '#urlcheck'

Can see silences on prometheus but can't display alarms on alertmanager;

enter image description here

enter image description here

markalex
  • 8,623
  • 2
  • 7
  • 32
tcblue
  • 179
  • 3
  • 12
  • Please replace the images with the text sources. You may embed code into the question by surrounding it with three backtick delimeters (```) – DazWilkin May 08 '22 at 17:23
  • Can the Prometheus server access the Alertmanager? Can you browse the Prometheus server's targets and get success scraping the Alertmanager metrics? – DazWilkin May 08 '22 at 17:37
  • @dazwilkin hi i have update as text the codes. Alertmanager and prometheus on same server. Alertmanager on 9093 port as default prometheus is on 9090 port. – tcblue May 09 '22 at 17:28
  • @DazWilkin I can view the silences rules I created on the alertmanager on prometheus, but the alarms cannot be displayed on the alertmanager. So I am able to successfully connect to alertmanager via prometheus. I have added the screenshots to topic. – tcblue May 09 '22 at 17:39
  • Are you running the both processes as containers? From what I understand of your config, Prometheus is working correctly. But, I suspect, it's unable to reach Alertmanager. The Alertmanager process is working correctly but it's not receiving data from Prometheus. I wonder whether Prometheus is unable to access Alertmanager. Even if both running locally, the Prometheus container would need to be able to access the Alertmanager endpoint and, if both containerized, could not do this unless the Alertmanager port is published to the host. – DazWilkin May 09 '22 at 17:51
  • Can Prometheus scrape Alertmanager's metrics? – DazWilkin May 09 '22 at 17:52
  • yes it can, i can display alertmanager's metrics on prometheus. I sending post request to alertmanager with manually /api/v1/alerts [{"labels":{"EndpointDown":"TestAlert1"}}] and it successfully send alert to channel. But still i cant see alerts on alertmanager. – tcblue May 09 '22 at 18:30
  • Of course, sorry you showed that. Yes. Hmm, I'm stumped. – DazWilkin May 09 '22 at 19:14

3 Answers3

0

default Prometheus config for alertmanager is not correct

use this config to connecting your alertmanager to Prometheus

alerting:
  alertmanagers:
  - static_configs:
    - targets: ['127.0.0.1:9093']

then sudo systemctl status prometheus.service

and check status with sudo systemctl status prometheus.service

Moein T
  • 86
  • 1
  • 4
0

The issue is with the version of alertmanager u r are using. To validate you can go to toolbar status > Runtime & Build Information in prometheus and check for Alertmanagers endpoint on the page. If the endpoint has v2 in it like /api/v2/alerts, then you are supposed to use latest version of alertmanager.

You can also run this expression in prom to validate the above,

rate(prometheus_notifications_errors_total[5m]) / rate(prometheus_notifications_sent_total[5m]) > 0.03
Shubham Negi
  • 64
  • 4
0

Use the default Prometheus.yml Configuration instead of that:

Alertmanager configuration

    alerting:
      alertmanagers:
        - static_configs:
          - targets:
              - localhost:9093

Then restart your prometheus.yml file from Terminal it will definetely show you the alerts in Alert Manager.

https://github.com/prometheus/alertmanager

Kartikey Srivastava
  • 11
  • 3