I am using prometheus from kube prom stack. Version of Prometheus is 2.36. I have configured the values.yaml file for prometheus and also after the deployment, i can observe the cofiguration of retention size and scrape interval etc as below:
kubectl -n monitoring get prometheus abc-kube-pro-prometheus -oyaml | grep -i retention retention: 30m
retentionSize: 512MB
kubectl -n monitoring get prometheus abc-kube-pro-prometheus -oyaml | grep -i storage storage:
storage: 1Gi
storageClassName: XXX-sc-statefulset
But, when i checked the prometheus disk usage, it shows more than 512 MB.
$df -h /dev/sdd 970.0M 950.0M 0 100% /prometheus
WAL Size
$ du -h ./wal 889.4M ./wal
It means it is ignoring the retentionSize of 512 MB. I think this issue was fixed in earlier version of prometheus. I also waited for 2-3 hours for compaction to kick , but i didnt see any change.
Am i missing any configuration?
retention: 30m
scrapeInterval: 1s
retentionSize: 512MB
resources:
requests:
storage: 1Gi
