-1

I have a kubernetes cluster which running on EKS

I already installed prometheus and alertmanager from this repository.

This is the set of tools, so I think it is useful for a begginner like me.

Now I can see prometheus dashboad via localhost:9090 and my pods are here.

There are somehow three alertmanager.(Maybe same as node?)

$kubectl get pod -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   aws-node-669m7                         1/1     Running   0          29m
kube-system   aws-node-7rwhg                         1/1     Running   0          29m
kube-system   aws-node-f7tnf                         1/1     Running   0          39m
kube-system   cluster-autoscaler-78c99d7c59-6gbn7    1/1     Running   0          31m
kube-system   coredns-7dbf6bcd5b-5hgjx               1/1     Running   0          49m
kube-system   coredns-7dbf6bcd5b-ff5b4               1/1     Running   0          49m
kube-system   kube-proxy-jrwxh                       1/1     Running   0          29m
kube-system   kube-proxy-lwm9q                       1/1     Running   0          39m
kube-system   kube-proxy-xp9l9                       1/1     Running   0          29m
monitoring    alertmanager-main-0                    2/2     Running   0          35m
monitoring    alertmanager-main-1                    2/2     Running   0          35m
monitoring    alertmanager-main-2                    2/2     Running   0          35m
monitoring    blackbox-exporter-69f4d86566-d99bk     3/3     Running   0          36m
monitoring    grafana-79cd8d4b69-2k9np               1/1     Running   0          36m
monitoring    kube-state-metrics-56f8746666-q298b    3/3     Running   0          31m
monitoring    node-exporter-ndln7                    2/2     Running   0          29m
monitoring    node-exporter-psqng                    2/2     Running   0          29m
monitoring    node-exporter-z92vx                    2/2     Running   0          35m
monitoring    prometheus-adapter-77f56b865b-4sgv7    1/1     Running   0          31m
monitoring    prometheus-adapter-77f56b865b-f9tx6    1/1     Running   0          35m
monitoring    prometheus-k8s-0                       2/2     Running   0          35m
monitoring    prometheus-k8s-1                       2/2     Running   0          35m
monitoring    prometheus-operator-54f6687b97-5mdbb   2/2     Running   0          35m

then, What I want to do is set the alert which send the email to my gmail account when nodes are more than 4.

so I made the alertmanager.yml (It is not the alert what I meant, this is just the sample, just practicing.)

global:
  resolve_timeout: 5m
  smtp_from: 'prometheus.test@example.com'
  smtp_smarthost: 'localhost:25'
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
  routes:
  - receiver: 'mailtest'
    match:
      test_type: SampleAlert
receivers:
- name: 'mailtest'
  email_configs:
    - to: 'root@localhost.localdomain'
      require_tls: false
- name: 'web.hook'
  webhook_configs:
  - url: 'https://127.0.0.1:5001/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Then document says, I can attache this as nohup ./alertmanager --config.file=alertmanager.yml &

however, I don't have alertmanager command in my local PC.

How should I apply this yml to the alertmanager pods running on EKS?

whitebear
  • 11,200
  • 24
  • 114
  • 237

1 Answers1

1

As mentioned in docs: You can add Alert Manager configuration in your values.yaml file under values.alertmanager.config Alert manager is imported to kubernetes as a secret, if you want to se what is being applied, you need to run kubectl get secrets then extract data from it.

To do more complex manipulations on your configurations, you need to get familiar with libsonnet .

Aref Riant
  • 582
  • 3
  • 14
  • Thank you very much!, I read the document, and maybe I god a first clue... – whitebear May 02 '23 at 11:34
  • I made `my-kube-prometheus` as document says , then I compiled files under `manifest` with `build.sh`. I am learning step by step. but, `values.yaml file under values.alertmanager.config` what does it mean? There is no values.yaml file and I couldn't find `values.alertmanager.config` even floder search. – whitebear May 02 '23 at 11:37
  • I made the `alert-example.jsonnet` with this sample. `https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/alertmanager-configuration.md` , then `sh build.sh alert-example.jsonnet` – whitebear May 02 '23 at 11:57
  • However it made the wrong yml..... – whitebear May 02 '23 at 11:58
  • you can add configuration to values in the jsonnet file you are intending. – Aref Riant May 02 '23 at 12:22
  • bear in mind that you can use kube-prometheus-stack [https://github.com/prometheus-operator/kube-prometheus] for easier and more automated operations. – Aref Riant May 02 '23 at 12:23
  • THank you I will use that library as well, IT is still unclear for me, but thanks to your help, I can step forward. – whitebear May 03 '23 at 13:02