Kubernetes HPA on AKS is failing with error 'missing request for cpu'

Question

I am trying to setup HPA for my AKS cluster. Following is the Kubernetes manifest file:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: XXXXXX\tools\kompose.exe
      convert
    kompose.version: 1.21.0 (992df58d8)
  creationTimestamp: null
  labels:
    io.kompose.service: loginservicedapr
  name: loginservicedapr
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: loginservicedapr
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: XXXXXX\kompose.exe
          convert
        kompose.version: 1.21.0 (992df58d8)
      creationTimestamp: null
      labels:
        io.kompose.service: loginservicedapr
    spec:
      containers:          
        image: XXXXXXX.azurecr.io/loginservicedapr:latest
        imagePullPolicy: ""
        name: loginservicedapr
        resources:
          requests:
            cpu: 250m
          limits:
            cpu: 500m            
        ports:
        - containerPort: 80
        resources: {}
      restartPolicy: Always
      serviceAccountName: ""
      volumes: null
status: {}
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: XXXXXXXXXX\kompose.exe
      convert
    kompose.version: 1.21.0 (992df58d8)
  creationTimestamp: null
  labels:
    io.kompose.service: loginservicedapr
  name: loginservicedapr
spec:
  type: LoadBalancer
  ports:
  - name: "5016"
    port: 5016
    targetPort: 80
  selector:
    io.kompose.service: loginservicedapr
status:
  loadBalancer: {}

Following is my HPA yaml file:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:  
  name: loginservicedapr-hpa
spec:
  maxReplicas: 10 # define max replica count
  minReplicas: 3  # define min replica count
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: loginservicedapr
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
  - type: Pods
    pods:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

But when HPA is failing with the error 'FailedGetResourceMetric' - 'missing request for CPU'.

I have also installed metrics-server (though not sure whether that was required or not) using the following statement: kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml

But still I am getting the following output when I do 'kubectl describe hpa':

Name:                                                  loginservicedapr-hpa
Namespace:                                             default
Labels:                                                fluxcd.io/sync-gc-mark=sha256.Y6dHhIOs-hNYbDmJ25Ijw1YsJ_8f0PH3Vlruj5rfbFk
Annotations:                                           fluxcd.io/sync-checksum: d5c0d9eda6db0c40f1e5e23e1356d0268dbccc8f
                                                       kubectl.kubernetes.io/last-applied-configuration:
                                                         {"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{"fluxcd.io/sync-checksum":"d5c0d9eda6db0c40f1e5...
CreationTimestamp:                                     Wed, 08 Jul 2020 17:19:47 +0530
Reference:                                             Deployment/loginservicedapr
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          3
Max replicas:                                          10
Deployment pods:                                       3 current / 3 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: missing request for cpu
Events:
  Type     Reason                        Age                      From                       Message
  ----     ------                        ----                     ----                       -------
  Warning  FailedComputeMetricsReplicas  33m (x1234 over 6h3m)    horizontal-pod-autoscaler  Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: missing request for cpu
  Warning  FailedGetResourceMetric       3m11s (x1340 over 6h3m)  horizontal-pod-autoscaler  missing request for cpu

I have 2 more services that I have deployed along with 'loginservicedapr'. But I have not written HPA for those services. But I have included resource limits for those services as well in their YAML files. How to make this HPA work?

In your Deployment duplicated the resources section, Just remove the ```resources {}``` part — Dashrath Mundkar, Jul 08 '20 at 18:29

score 6 · Answer 1 · answered Jul 08 '20 at 18:25

6

resources appears twice in your pod spec.

        resources:         # once here
          requests:
            cpu: 250m
          limits:
            cpu: 500m            
        ports:
        - containerPort: 80
        resources: {}      # another here, clearing it

answered Jul 08 '20 at 18:25

jbielick

2,800
17
28

1

I removed it but still having the issue. – Sormita Chakraborty Jul 09 '20 at 04:16

score 5 · Answer 2 · answered Jul 09 '20 at 04:38

5

I was able to resolve the issue by changing the following in my kubernetes manifest file from this:

resources:
          requests:
            cpu: 250m
          limits:
            cpu: 500m

to the following:

resources:
          requests:
            cpu: "250m"
          limits:
            cpu: "500m"

HPA worked after that. Following is the GitHub link which gave the solution: https://github.com/kubernetes-sigs/metrics-server/issues/237 But I did not add any Internal IP address command or anything else.

answered Jul 09 '20 at 04:38

Sormita Chakraborty

1,015
2
19
36

1

Your before and after are same. k8s will treat both `250m` and `"250m"` as string. Problem was with duplicate `resources` block (as others have mentioned) – nakamume Apr 09 '21 at 10:54
Thanks this took me two hours to figure out then I put in the "" and it worked so annoying. – Jason Nov 28 '22 at 13:41

score 2 · Answer 3 · answered Jul 08 '20 at 20:12

2

This is typically related to the metrics server.

Make sure you are not seeing anything unusual about the metrics server installation:

# This should show you metrics (they come from the metrics server)
$ kubectl top pods
$ kubectl top nodes

or check the logs:

$ kubectl logs <metrics-server-pod>

Also, check your kube-controller-manager logs for HPA events related entries.

Furthermore, if you'd like to explore more on whether your pods have missing requests/limits you can simply see the full output of your running pod managed by the HPA:

$ kubectl get pod <pod-name> -o=yaml

Some other people have had luck deleting and renaming the HPA too.

answered Jul 08 '20 at 20:12

Rico

58,485
12
111
141

I checked the output of $ kubectl get pod -o=yaml and resource and limit are both present. – Sormita Chakraborty Jul 09 '20 at 04:20
1

I ran the following 'kubectl logs -n kube-system -l k8s-app=metrics-server --container metrics-server' for which I got the output:unable to fetch pod metrics for pod default/loginservicedapr-67fdbdcb59-vklb9: no metrics known for pod-how to fix this – Sormita Chakraborty Jul 09 '20 at 04:28

score 2 · Answer 4 · answered Mar 06 '23 at 12:47

2

I had this problem because I still had a terminated pod that did not specify the cpu resource

I.e. the yaml was:

resources: {}

instead of:

resources:
  requests:
    cpu: 250m

answered Mar 06 '23 at 12:47

veuncent

1,599
1
20
17

Kubernetes HPA on AKS is failing with error 'missing request for cpu'

4 Answers4