Kubernetes autoscaling not working - showing unkown

Question

There are a lot of questions about this topic and I tried a lot of stuff but it's still not working.

I'm new to Kubernetes. I have a Kubernetes Cluster with 2 Nodes (1 PC, 1 VM on a different PC), 1 Master, 1 Node. I started a website (Docker Image) and the Pod is running on the Node (not master). Now I wanted to autoscale the Pod, this is what I did:

Set --request='cpu=50m' to the pod
Created hpa: kubectl autoscale deployment testwebsite --min=1 --max=4 --cpu-percent=25
Used: kubectl get hpa -w

And now there is the problem that the output shows as current value. I read a lot that I have to assign the request to the pod, as I did. Checked it via: kubectl get pod testwebsite --out=yaml

I also created the metrics-server version 1.8+. Made sure it's running via: kubectl get pods --all-namespaces

After that I tried to debug the HPA and looked into it via:

kubectl describe hpa testwebsite

That showed me this:

Name:                                                  testwebsite
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Thu, 04 Apr 2019 14:08:57 +0200
Reference:                                             Deployment/testwebsite
Metrics:                                               ( current / target )   resource cpu on pods  (as a percentage of request):  <unknown> / 25%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
 AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
 ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
  Type     Reason                        Age                   From                       Message
  ----     ------                        ----                  ----                       -------
  Warning  FailedComputeMetricsReplicas  7m24s (x12 over 10m)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedGetResourceMetric       5s (x41 over 10m)     horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API

After 10+ mins it's not showing the current value and it won't scale up even when the cpu has 50-100% usage.

As I said I'm new to Kubernetes and I really hope someone can help me.

Best regards,

Nico aka Myridor

EDIT: Using Ubuntu 18.04 LTS on both Nodes | Kubernetes Version: 1.14.0

score 0 · Answer 1 · answered Apr 05 '19 at 04:13

0

are your sure on kubernetes metrics server & metrics api is running? which is necessary first to and it use to check cpu usage and limits

You can remove the LIMITS from your deployments if any there and try it.

answered Apr 05 '19 at 04:13

Harsh Manvar

27,020
6
48
102

If I use `kubectl get pods --all-namespaces` it says that the metrics server is running 1/1 – Myridor Apr 05 '19 at 06:45

score 0 · Accepted Answer · answered Apr 05 '19 at 14:14

The issue is not in HPS itself but in metrics server, which is not able to scrape metrics.

I recreated the issue by cloning metrics-server git and then creating it by kubectl create -f deploy/1.8+/.

Next I edited the metrics server deployment by running: kubectl edit deployment metrics-server -n kube-system

Under spec: -> containers: add following flag:

spec:
      containers:
      - command:
        - /metrics-server
        - --kubelet-insecure-tls

As described in git:

--kubelet-insecure-tls: skip verifying Kubelet CA certificates. Not recommended for production usage, but can be useful in test clusters with self-signed Kubelet serving certificates.

After the deployment is updated and metrics server scraped the metrics you should see HPA updated with current targets. Default scrape interval is 60s, you can change it by adding following flag to the above --metric-resolution=10s <- interval set to 10s.

Kubernetes autoscaling not working - showing unkown

2 Answers2