0

We are using prometheus adapter to allow for custom metric auto scaling in k8s. We have everything setup where the HPA is getting the metric but it is exhibiting improper scale up even when the metric is below the target.

Polling the metric api server we can see the correct metric. enter image description here

We can see the metric in prometheus over time where value never gets above 25% but scaling of the container still happens from 1-4. enter image description here

We can see where the pod has a target value of 100 but has shown history of scaling from 1 to 4. enter image description here

And the config for rule in prometheus adapter. enter image description here

Pod Helm Config enter image description here

Any help or direction to check next would be great. The expected result is we should not see any autoscaling as the target, 100, is always greater then the metric.

UPDATE 1: Ran on watch and got this enter image description here

with this coming from prometheus -v 8 enter image description here enter image description here

user2434674
  • 520
  • 1
  • 8
  • 19
  • Running Kube v1.11.8 – user2434674 Sep 09 '19 at 02:35
  • Can you check if the hpa actually reports target as > 100 when the rescale happens? You could watch for `kubectl get hpa -w` output f.i. – shkaper Sep 09 '19 at 03:54
  • You can also check prometheus adapter logs (make sure verbosity is at least `--v=6`). Look for the request made by the HPA and the actual prometheus query it results in. This query might or might not be different from what you're querying in your grafana screenshot. – shkaper Sep 09 '19 at 04:00
  • @shkaper See updated post. – user2434674 Sep 09 '19 at 04:39
  • **(1.)** Can you check that when the HPA scaling events happen (below target/above target), the metric is really below/above 100 (query from Custom Metrics API with `kubectl get --raw="..."`)? **(2.)** Can you manually trigger the metric to go up and down to observe the reaction of the HPA in detail? **(3.)** What does the metric mean? Why is it associated with a Namespace object? What component produces the metric? – weibeld Sep 09 '19 at 10:02

0 Answers0