How do I verify a configuration change on a bitnami/kube-prometheus deployment?

Question

I have an existing deploy of bitnami/kube-prometheus deployed via :

helm upgrade --install prometheus -n monitoring -f .\input\prom-values.yaml bitnami/kube-prometheus

I have made some changes to prom-values.yaml and did a corresponding 'helm upgrade' but I cannot seem to find a straightforward way of verifying the new configuration.

This is prom-values.yaml before the changes :

prometheus:
persistence:
    enabled: true
    size: 10Gi

alertmanager:
  persistence:
    enabled: true
  
node-exporter:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: NoSchedule

I have verified the changes via :

helm template prometheus bitnami/kube-prometheus --namespace monitoring --version 8.14.0 -f prom-values.yaml > ./output/values.yaml

And this is what prom-values.yaml looks like with the changes :

prometheus:
persistence:
    enabled: true
    size: 10Gi
  ruleSelector:
    matchLabels:
      release: prometheus

alertmanager:
  persistence:
    enabled: true
  configSelector:
    matchLabels:
      release: prometheus

node-exporter:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: NoSchedule

According to the documentation I am following I should be able to verify the new configuration these two commands:

kubectl get prometheuses.monitoring.coreos.com -o yaml
kubectl get alertmanagers.monitoring.coreos.com -o yaml

Presently both commands yield no output (I dont know why exactly):

apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I have tried other ways of verifying the configuration but Im not sure if I am doing it the correct way. I sent a POST request to the /reload/ endpoint manually via :

curl -X POST http://prometheus-kube-prometheus-prometheus.monitoring:9090/-/reload

After running this the Prom UI shows that the configuration was reloaded successfully :

However when I check the status of the pod I do not see any pod restarts. All pods have a restart count of 0.

My issues :

How does the config-reloader for bitnami/kube-prometheus really work ? Does it initiate a pod restart or it runs silently in the background. Where can I see its logs ? :
How and where can I exactly check if the config was applied successfully. When I kubectl exec into the prometheus pod and navigate to /conf/prometheus.yml I do not see the changes. Is this the correct file ?
Where is the configuration for the alert manager component and does it also allow to see the kind of changes I have made. When I kubectl exec into the alert manager pod and navigate to /conf/config I see one file ( alertmanager.yaml.gz) but I can neither inspect it or copy to local workstation :

score 0 · Answer 1 · answered Jul 14 '23 at 11:11

They are two ways to verify config changes :

From AlertManager pod logs

kubectl -n monitoring logs alertmanager-prometheus-kube-prometheus-
alertmanager-0 -c config-reloader
level=info ts=2023-06-26T06:59:24.380805173Z caller=main.go:115 msg="Starting prometheus-config-reloader" version="(version=0.66.0, branch=, revision=unknown)"
cfg=/etc/alertmanager/config/alertmanager.yaml.gz out=/etc/alertmanager/config_out/alertmanager.env.yaml dirs=
level=info ts=2023-07-07T09:00:11.282196739Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/alertmanager/config/alertmanager.yaml.gz cfg_out=/etc/alertmanager/config_out/alertmanager.env.yaml watched_dirs=

The msg="Reload triggered" shows that the reload occured. If it fails it should also show error message.

From the Prometheus UI

kubectl -n monitoring port-forward service/prometheus-kube-prometheus-prometheus 9090

Go to Status >> Runtime and Build Information

How do I verify a configuration change on a bitnami/kube-prometheus deployment?

1 Answers1