3

I've deployed kube-prometheus-stack and default rules / alerts helped me a lot, but I need to create custom ones. Specifically, I would like to get information when my pods are 50% (warning) or 70% (critical) over their limits. For example, if limits are for my pod 1 CPU / 1 GB of RAM, I would like to fire alerts if the pod is using 1.5 vCPU or 1.5 GB of RAM

I know this requires combinining kube_pod_container_resource_limits_cpu_cores and kube_pod_container_resource_limits_memory_bytes along with something like node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate for current CPU usage of container, or node_namespace_pod_container:container_memory_working_set_bytes but I don't have an idea how to divide these two metrics and get end result where the current usage is over limits

For setting up 50% / 70%, I would of course create two different alerts with different severity, but the PromQL statement is bugging me. Thanks a lot in advance for your help!

Rookie
  • 83
  • 1
  • 8
  • 1
    Perhaps you want to see if a Pod uses more recources than it __*requests*__? Because it's not going to use more that the limit. Because, you know... it's a limit. As for the answer, to divide one metric by another you need to use a common label they both have. For example: `pod_memory_usage / on(pod_name) pod_memory_limit`, 'pod_name' in the example is the common label. – anemyte Apr 08 '21 at 19:39

0 Answers0