1

My environment:

  • kubernetes 1.8.4 on AWS, deployed with kops.
  • Heapster 1.5.0 with influxdb sink and 60s metric resolution

When my pods are idle or low traffic everything's fine. My HPAs can get data out of heapster and I can see data on grafana, pulling it out from influxdb.

When I start loadtesting a pod (putting it under some traffic, starting at 10 rqs/second) i stop getting info about the CPU usage in grafana and HPAs start getting this:

Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedGetResourceMetric       2m (x13 over 3h)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from heapster
  Warning  FailedComputeMetricsReplicas  2m (x13 over 3h)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster

After the load finishes, I get almost immediately back my CPU metrics both in influxdb and HPAs. Please note that, during the same period, I never ever lose data about memory usage.

Any help with troubleshooting and/or solving this would be very appreciated.

For the records, I also posted an issue on heapster's github: https://github.com/kubernetes/heapster/issues/1937

whites11
  • 12,008
  • 3
  • 36
  • 53
  • I'm seeing the same issue, did you ever find a solution? Was it just that the node was starved of CPU and couldn't collect the metrics? – GreenKiwi May 25 '18 at 05:35
  • Unluckily not, still having the same problem. Also opened a ticket on GitHub in the heapster project but got no solution. On k8s 1.9+ autoscaling doesn't use heapster anymore, but I didn't try the same load testing with the new system – whites11 May 25 '18 at 05:38

0 Answers0