1

I used the following metrics inside hpa

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: app-svc-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: app-svc
  minReplicas: 1
  maxReplicas: 1000
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
  - type: Pods
    pods:
      metric:
        name: packets-per-second
      target:
        type: AverageValue
        averageValue: 1k

But the hpa is unable to get the metrics

Warning FailedGetPodsMetric 14s (x6 over 1m) horizontal-pod-autoscaler unable to get metric packets-per-second: unable to fetch metrics from custom metrics API: the server could not find the descriptor for metric custom.googleapis.com/packets-per-second: googleapi: Error 403: Permission monitoring.metricDescriptors.get denied (or the resource may not exist)., forbidden

I am running the pods on a dedicated nodepool and each nodes is running under a service account.

The service account does have these iam roles

Monitoring Viewere, Monitoring Metrics Writer

Unsure how to fix this error. Any pointers are greatly appreciated. Thanks.

Varunkumar Manohar
  • 887
  • 3
  • 11
  • 29
  • you mentioned IAM role, is it running on EKS? did you try to deploy with no serviceaccount? did you add the right privileges to the metric server? – Mickey Hovel Apr 09 '20 at 16:36
  • it is running on Google Cloud Kubernetes Engine – Varunkumar Manohar Apr 09 '20 at 16:48
  • Are you using RBAC in the current configuration? – Mickey Hovel Apr 09 '20 at 16:49
  • No I am not using RBAC – Varunkumar Manohar Apr 09 '20 at 16:50
  • Have you checked the metric server logs? – Mickey Hovel Apr 09 '20 at 16:52
  • I ran the same hpa on a new cluster running under default service account and I see this new error unable to get metric packets-per-second: unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered Any pointers ? – Varunkumar Manohar Apr 09 '20 at 16:53
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/211305/discussion-between-varunkumar-manohar-and-mickey-hovel). – Varunkumar Manohar Apr 09 '20 at 16:59
  • 1
    The error that you are encountering points to the either permission denied or lack of metric named `packets-per-second`. There is a GCP guide with very similar configuration like yours: [Cloud.google.com: HPA](https://cloud.google.com/kubernetes-engine/docs/how-to/horizontal-pod-autoscaling). It mentions that `packets-per-second` is a custom metric that needs to be created. Did you create this metric? Can you see this metric in `GCP->Monitoring->Metrics Explorer`? – Dawid Kruk Apr 10 '20 at 12:57
  • Thank you that answers my question and there was yet another issue which I managed to find out why. I will post my answer below – Varunkumar Manohar Apr 10 '20 at 17:09

1 Answers1

1

I had a cluster with workload identity enabled. Apparently when a cluster has workload identity enabled, the metrics fetch was failing.

1) I had to install the custom stack driver adapter and create the custom metric as pointed by David Kruk in his comments

2) I had to add the hostNetwork:true in the custom stackdriver adapter deployment pod spec. The issue is mentioned here in github repository for csa

With these two updates, the autoscaler works as expected.

Varunkumar Manohar
  • 887
  • 3
  • 11
  • 29