2

I am trying to deploy HPA for my application but I have been getting the Unauthorized error

kubectl desribe hpa homekitchen:
Events:
  Type     Reason                   Age                   From                       Message
  ----     ------                   ----                  ----                       -------
  Warning  FailedGetResourceMetric  13s (x488 over 130m)  horizontal-pod-autoscaler  unable to get metrics for resource memory: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=app%3Dhomekitchen\": Unauthorized") has prevented the request from succeeding (get pods.metrics.k8s.io)

kubectl get hpa:

NAME          REFERENCE                TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
homekitchen   Deployment/homekitchen   <unknown>/75%, <unknown>/85%   2         3         2          169m

My HPA config:

---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: homekitchen
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: homekitchen
  minReplicas: 2
  maxReplicas: 4
  behavior:
    scaleUp:
      policies:
        - type: Pods
          value: 1
          periodSeconds: 70
    scaleDown:
      policies:
        - type: Percent
          value: 10
          periodSeconds: 70
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 85
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 75

In the deployment config file, I have added the following

 resources:
          requests:
            memory: 500Mi
            cpu: 500m
          limits:
            memory: 1500Mi
            cpu: 1500m

And my metrics server (namespace = kube-system) logs show following error:

I0717 11:49:36.396129       1 server.go:188] "Failed probe" probe="metric-storage-ready" err="not metrics to serve"
I0717 11:49:46.396613       1 server.go:188] "Failed probe" probe="metric-storage-ready" err="not metrics to serve"
E0717 11:49:49.529809       1 scraper.go:139] "Failed to scrape node" err="GET \"https://**.**.**.**:**/stats/summary?only_cpu_and_memory=true\": bad status code \"403 Forbidden\"" node="gke-******"
E0717 11:49:49.529921       1 scraper.go:139] "Failed to scrape node" err="GET \"https://**.**.**.**:***/stats/summary?only_cpu_and_memory=true\": bad status code \"403 Forbidden\"" node="gke-*****"

ClusterRole for metrics server:

Name:         system:metrics-server
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              kubernetes.io/cluster-service=true
Annotations:  <none>
PolicyRule:
  Resources         Non-Resource URLs  Resource Names  Verbs
  ---------         -----------------  --------------  -----
  deployments.apps  []                 []              [get list update watch]
  namespaces        []                 []              [get list watch]
  nodes             []                 []              [get list watch]
  pods              []                 []              [get list watch]
                    [/metrics]         []              [get]

With my limited knowledge, I am unable to figure out why it's getting unauthorized errors.

Please help me out guys

Thanks in advance

Pranav
  • 95
  • 7
  • There is a similar case with an answer. Could you check if it could be a solution for your case also: https://stackoverflow.com/questions/61125754/hpa-cannot-get-metrics-due-to-403-errors Could you also check if your metrics-server has recent version if you installed it manually. – VAS Jul 28 '21 at 09:59

0 Answers0