GCP - Scale GKE pods based on custom logging metric using HPA

Question

I want to use the Custom log metrics on GKE HPA. Metrics are able to view on metrics explorer but unable to use it on HPA . We have installed Custom metrics adapter and We are able to use other custom metrics like kubernetes.io|pod|network|received_bytes_count successfully for scaling. Below image shows the Metrics explorer graph for custom metric that i want to use on HPA

This metric was created from application logs

Used following HPA yaml to use that metric

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name:   "similar-products-rts-hpa"
  namespace: relevancy
spec:
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 120
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: similar-products
  minReplicas: 3
  maxReplicas: 6
  metrics:
    - type: Pods
      pods:
        metric:
          name: "logging.googleapis.com|user|Similar_Products(RTS)_Inbound_Request_Count"
        target:
          type: AverageValue
          averageValue: 25

Please find the error below

The HPA was unable to compute the replica count: unable to get metric logging.googleapis.com|user|Similar_Products(RTS)_Inbound_Request_Count: unable to fetch metrics from custom metrics API: googleapi: Error 400: The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series., badRequest

Which type of error are you getting? Are you deploying the application with metrics? Are you following a document? — Nadia Espinosa, Oct 05 '21 at 23:08

Gari Singh · Accepted Answer · 2021-10-06T10:43:48.933

1

Unfortunately, upper case letters in metric names are not supported as HPA treats metrics as pseudo resources which means they are not case sensitive. I also believe that parentheses are invalid characters as well for metric names.

Any chance you can change your metric name to lowercase and remove the parentheses? Maybe something like similar_products_rts_inbound_request_count?

EDIT: The other issue I just noticed is that the metric is a container metric and not a pod metric. Prior to this change, it was necessary to modify the custom metrics adapter deployment to support container metrics. You can either update your deployment with the current manifest,or you can modify your current deployment by adding --fallback-for-container-metrics=true:

spec:
      serviceAccountName: custom-metrics-stackdriver-adapter
      containers:
      - image: gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.0-gke.0
        imagePullPolicy: Always
        name: pod-custom-metrics-stackdriver-adapter
        command:
        - /adapter
        - --use-new-resource-model=true
        - --fallback-for-container-metrics=true
        resources:
          limits:
            cpu: 250m
            memory: 200Mi
          requests:
            cpu: 250m
            memory: 200Mi

edited Oct 06 '21 at 10:43

answered Oct 06 '21 at 08:21

Gari Singh

11,418
2
18
41

I can change the metric name to lower case is that the reason for time series error – sankar Oct 06 '21 at 09:09
That should resolve your issue. You may also need to remove the parentheses from the metric name as well. I updated my answer above. – Gari Singh Oct 06 '21 at 09:10
The HPA was unable to compute the replica count: unable to get metric logging.googleapis.com|user|similarproducts_inbound_request_count: unable to fetch metrics from custom metrics API: googleapi: Error 400: The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series., badRequest still getting same error – sankar Oct 06 '21 at 09:45
When did you deploy the custom metrics adapter? – Gari Singh Oct 06 '21 at 09:58
few months back – sankar Oct 06 '21 at 09:59
This metric is not an external metric this is stack driver log based metric .Look the shared image again. Dont know whether it is possible to use this metric under scaling – sankar Oct 06 '21 at 10:02
This should be possible. I just noticed that your metric is a container metric. I edited my answer above to account for this as well. – Gari Singh Oct 06 '21 at 10:44
asked my cloud support team to update the adapter i would test this and let you know your answer helped on this. Thanks for your time on this much Appreciated – sankar Oct 08 '21 at 07:14
It worked thanks for help – sankar Oct 19 '21 at 18:56
above stack driver adapter update didnt helped still getting same error again The HPA was unable to compute the replica count: unable to get metric logging.googleapis.com|user|similarproducts_inbound_request_count: unable to fetch metrics from custom metrics API: googleapi: Error 400: The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series., badRequest – sankar Oct 25 '21 at 11:35

GCP - Scale GKE pods based on custom logging metric using HPA

1 Answers1