-1

docker-compose.yml: Service is up and running.

version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus/conf:/etc/prometheus
#      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alertmanager/alert_rules.yml:/alertmanager/alert.rules.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    ports:
      - 9093:9093
    volumes:
      - ./alertmanager/:/etc/alertmanager/
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'

prometheus.yml: Prometheus config file with targets and alerts target sets. The alertmanager target url is working fine.

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 10.10.5.14:9093
    scheme: http
    timeout: 10s
    api_version: v1
rule_files:
  - "/alertmanager/alert.rules:/alertmanager/alert.rules"
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - "google.com"
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

alert.rules: Standard alert rule.

groups:
- name: example
  rules:

  # Alert for any instance that is unreachable for >1 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 1m

alertmanager.yml: Code to send alerts via gmail smtp.

route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-me

receivers:
- name: email-me
  email_configs:
  - to: anyemail.com
    from: senderlogin@gmail.com
    smarthost: smtp.gmail.com:587
    auth_username: "senderlogin@gmail.com"
    auth_identity: "senderlogin@gmail.com"
    auth_password: "password"

Problem: All containers are working but there are no rule in prometheus rules. And the letters do not come accordingly either. I can’t understand what I missed. Any hints are welcome. Thanks. enter image description here enter image description here

offline
  • 25
  • 1
  • 8

1 Answers1

0

Alertmanager is not on localhost from the view of the prometheus container, the networking is namespaced in containers so you need to contact alertmanager using DNS discovery to find the current container IP. With compose, DNS is automatically configured for the service name, in this case alertmanager:

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager:9093
BMitch
  • 231,797
  • 42
  • 475
  • 450
  • I just changed the localhost. In fact, there is an external address, and i can go to the alert manager. – offline Jun 10 '20 at 13:14
  • Did you go to it from inside of the prometheus container? – BMitch Jun 10 '20 at 13:36
  • No, with external ip. Alertmanager container has an ip like 10.10.5.14:9093 – offline Jun 10 '20 at 13:46
  • It appears that you haven't verified that the prometheus container can reach that IP, and you also don't want to use the DNS name that would work for container to container networking. Not sure I can provide any further assistance at this point. – BMitch Jun 10 '20 at 14:08
  • My prometheus container has the same ip like alertmanager but port are different. They are in one network.I can not understand what the dns has to do with it. – offline Jun 10 '20 at 14:18
  • PING alertmanager from inside the prom container 64 bytes from 10.10.5.187: seq=0 ttl=64 time=0.062 ms – offline Jun 10 '20 at 14:29