0

We have multiple AWS accounts and network access is configured between two AWS accounts and service discovery is working with node-exporter. I have Prometheus configuration with some of the rules configured for the docker containers and now I have added one of the rules similar to the existing one to check if by mistakenly same container is launched in another AWS account and below is the rule. for exiting rules, {{ $labels.instance }} is printing in Alerts email, but not for the new rule which I have written newly

Scrape config for labels:

  - job_name: 'aws-conatiners'
    scheme: http
    ec2_sd_configs:
         - region: {{region}}
           port: 8181
    relabel_configs:
           - source_labels: [__meta_ec2_tag_Name]
             target_label: instance

The new rule which I have created to check if more than one container is running:

# Alert to check if more than one instance is running for backendapi service
  - alert: multiple_instances_are_running
    expr: sum(container_last_seen{name=~"backendapi"}) > 1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "More than one Instance (instance {{ $labels.instance }}) is running"
      description: "More than one Instance (instance {{ $labels.instance }}) is running for 5 minutes."

Can someone please check and help me to get the instance name printed in alert emails

KNCK
  • 103
  • 2
  • 12
  • Does this answer your question? [Missing labels in prometheus alerts](https://stackoverflow.com/questions/67273594/missing-labels-in-prometheus-alerts) – anemyte Mar 11 '22 at 13:48
  • Thanks, the link you have shared helped me a little bit to understand, I have changed my query to `sum((container_last_seen{name=~"backendapi"})) by (instance) > 1` but now I have one more question, how to get alerts based up on the job – KNCK Mar 11 '22 at 16:23
  • Sorry but I don't understand what exactly you want to do with the job, please elaborate. – anemyte Mar 11 '22 at 16:53

0 Answers0