How to get list of metrics available for HPA?

Question

I have GCP cluster which contains GKE application:

I want to scale application using HPA

Based on supporting for metrics

HPA able to read metrics from

metrics.k8s.io (resource metrics)
custom.metrics.k8s.io(custom metrics)
external.metrics.k8s.io(external metrics)

How could I check what metrics available? How could try this API on my own ? Is it possible at all?

P.S.

Based on suggested answer I executed command:

 kubectl get --raw https://MY-KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/default/pods

Response is:

{
  "items": [
    {
      "metadata": {
        "name": "prometheus-adapter-69fcdd56bc-2plh7",
        "namespace": "default",
        "selfLink": "/\r\napis/metrics.k8s.io/v1beta1/namespaces/default/pods/prometheus-adapter-69fcdd56bc-2plh7",
        "creationTimestamp": "2020-02-05T10:56:02Z"
      },
      "timestamp": "2020-02-05T10:55:22Z",
      "window": "30s",
      "containers": [
        {
          "name": "prometheus-adapter",
          "usage": {
            "cpu": "15\r\n31939n",
            "memory": "10408Ki"
          }
        }
      ]
    },
    {
      "metadata": {
        "name": "stackdriver-exporter-76fdbc9d8f-c285l",
        "namespace": "default",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/stackdriver-exporter-76fdbc9d8f-c285l",
        "creationTimestamp": "2020-0\r\n2-05T10:56:02Z"
      },
      "timestamp": "2020-02-05T10:55:22Z",
      "window": "30s",
      "containers": [
        {
          "name": "stackdriver-exporter",
          "usage": {
            "cpu": "79340n",
            "memory": "2000Ki"
          }
        }
      ]
    }
  ],
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods"
  }
}


$ kubectl top pods
NAME                                    CPU(cores)   MEMORY(bytes)
prometheus-adapter-69fcdd56bc-2plh7     2m           10Mi
stackdriver-exporter-76fdbc9d8f-c285l   1m           1Mi

But I still don't see all metrics available for HPA

Could you advise your steps/config? Did you use any tutorial? What types of metric you want to get? — PjoterS, Feb 05 '20 at 14:25

score 8 · Answer 1 · edited Feb 17 '21 at 20:33

Metrics server exposes metrics via below APIs.

/nodes - all node metrics; type []NodeMetrics
/nodes/{node} - metrics for a specified node; type NodeMetrics
/namespaces/{namespace}/pods - all pod metrics within namespace with support for all-namespaces; type []PodMetrics
/namespaces/{namespace}/pods/{pod} - metrics for a specified pod; type PodMetrics

You can view available metrics as below for example

$ kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "nodes",
      "singularName": "",
      "namespaced": false,
      "kind": "NodeMetrics",
      "verbs": [
        "get",
        "list"
      ]
    },
    {
      "name": "pods",
      "singularName": "",
      "namespaced": true,
      "kind": "PodMetrics",
      "verbs": [
        "get",
        "list"
      ]
    }
  ]
}

$ kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/default/pods
{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods"
  },
  "items": []
}

$ kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods
{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods"
  },
  "items": [
    {
      "metadata": {
        "name": "coredns-bcccf59f-jfl6x",
        "namespace": "kube-system",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-bcccf59f-jfl6x",
        "creationTimestamp": "2021-02-17T20:31:29Z"
      },
      "timestamp": "2021-02-17T20:30:27Z",
      "window": "30s",
      "containers": [
        {
          "name": "coredns",
          "usage": {
            "cpu": "1891053n",
            "memory": "8036Ki"
          }
        }
      ]
    },
    {
      "metadata": {
        "name": "coredns-bcccf59f-vmfvv",
        "namespace": "kube-system",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-bcccf59f-vmfvv",
        "creationTimestamp": "2021-02-17T20:31:29Z"
      },
      "timestamp": "2021-02-17T20:30:25Z",
      "window": "30s",
      "containers": [
        {
          "name": "coredns",
          "usage": {
            "cpu": "1869226n",
            "memory": "8096Ki"
          }
        }
      ]
    }
  ]
}

You can also use command kubectl top pods which internally calls the above API.

Custom Metrics

These are provided by adapters developed by vendors and what metrics are available will depend on the adapter. Once you know the metrics name You can use API to access it.

You can view available metrics as below and get the metrics name.

kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/custom.metrics.k8s.io/v1beta1

External Metrics

These are provided by adapters developed by vendors and what metrics are available will depend on the adapter. Once you know the metrics name You can use API to access it.

You can view available metrics as below and get the metrics name.

kubectl get --raw https://KUBE-APISERVER-IP:6443/apis/external.metrics.k8s.io/v1beta1

Edit:

You already have Prometheus adapter but if the metric is not exposed as custom metrics to be consumable by HPA then you need to expose the required metrics. Refer to this guide for this.

Is there snth like this ? https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.metricDescriptors/list?apix=true&apix_params=%7B%22name%22%3A%22projects%2Fbx-dev-contactmatch%22%2C%22filter%22%3A%22metric.type%20%3D%20%5C%22pubsub.googleapis.com%2Fsubscription%2Fnum_undelivered_messages%5C%22%20AND%20resource.labels.subscription_id%20%3D%20one_of(%5C%22partner-cmp-work%5C%22%2C%5C%22partner-cmp-work%5C%22)%22%7D — gstackoverflow, Feb 04 '20 at 18:24
You have a kubernetes cluster om GKE...have you ever connected to it using kubectl ? Run this to get a kubeconfig file gcloud container clusters get-credentials and then run kubectl config view to see the apiserver endpoint..for more details https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl — Arghya Sadhu, Feb 05 '20 at 03:09
I've executed query you mentioned but I see only 2 metrics - CPU and MEMORY. Could you explain this output ? — gstackoverflow, Feb 05 '20 at 11:11
CPU and Memory metrics is exposed by metrics server is available to HPA. For custom and external metrics you need consult the vendor providing the metrics — Arghya Sadhu, Feb 05 '20 at 15:38
I want to use this metric from promeheus: http://dl3.joxi.net/drive/2020/02/05/0005/3037/338909/09/3389e25477.jpg — gstackoverflow, Feb 05 '20 at 16:10
Its showing CPU metrics isn't that the system_cpu_usage? If not then you need to expose that metric..I have provided a reference in my answer on how to do that — Arghya Sadhu, Feb 05 '20 at 17:31
For `kubectl get --raw https://KUBE-APISERVER-IP:6443 /apis/external.metrics.k8s.io/v1beta1` I get `Error from server (NotFound): the server could not find the requested resource`. I have the metrics server installed. This is AWS. — Chris F, Feb 17 '21 at 20:19
Instead doing direct API requests you can use the built-in kubectlcommands. `kubectl api-resources --api-group=metrics.k8s.io` will return the supported metrics types. Then you can use `kubectl get TYPE.metrics.k8s.io -o yaml` to see what kind of metrics you have available. — Mr. Llama, Dec 03 '21 at 00:16

score 2 · Answer 2 · answered Feb 12 '20 at 12:14

On GKE case is bit different.

As default Kubernetes have some built-in metrics (CPU and Memory). If you want to use HPA based on this metric you will not have any issues.

In GCP concept:

Custom Metrics are used when you want to use metrics exported by Kubernetes workload or metric attached to Kubernetes object such as Pod or Node.
External Metrics- Metrics sent to Workspaces with a metric type beginning external.googleapis.com are known as external metrics. The metrics are typically exported by open-source projects and third-party providers. More details can be found here. Stackdriver Monitoring treats external metrics the same as custom metrics, with one exception. For external metrics, a resource_type of global is invalid and results in the metric data being discarded.

As GKE is integrated with Stackdriver

Google Kubernetes Engine (GKE) includes native integration with Stackdriver Monitoring and Stackdriver Logging. When you create a GKE cluster, Stackdriver Kubernetes Engine Monitoring is enabled by default and provides a monitoring dashboard specifically tailored for Kubernetes.

With Stackdriver Kubernetes Engine Monitoring, you can control whether or not Stackdriver Logging collects application logs. You also have the option to disable the Stackdriver Monitoring and Stackdriver Logging integration altogether.

Check Available Metrics

As you are using cloud environment - GKE, you can find all default available metrics by curiling localhost on proper port. You have to SSH to one of Nodes and then curl metric-server $ curl localhost:10255/metrics.

Second way is to check available metrics documentation.

IMPORTANT

You can see available metrics, however to use them in HPA you need deploy Adapters like Stackdriver adapter or Prometheus adapter. In default (version 1.13.11) GKE cluster you have already deployed metrics-server, heapster(deprecated in newer versions) and prometheus-to-sd-XXX. If you would like to use Stackdriver, you would have many config already applied, but if you want to use Prometheus you will need adjust Prometheus operators, adapters, deployments. Details can be found here.

On GKE docs, you can find some tutorials to use HPA with Custom Metrics or with HPA with External Metrics. You can also read about GKE monitoring with Prometheus and Stackdriver, depends on your needs.

As GKE is integrated with Stackdriver you can read article about Enabling Monitoring.

How to get list of metrics available for HPA?

P.S.

2 Answers2