0

Having configured metric rule in the prometheus-adapter, I can see resources as follows.

kubectl get --raw="/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "pods/istio_ai_retry_per_second",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "namespaces/istio_ai_retry_per_second",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

But when I try to get metrics values. The items are empty.

kubectl get --raw="/apis/custom.metrics.k8s.io/v1beta1/namespaces/aiservice-test/pods/*/istio_ai_retry_per_second" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/aiservice-test/pods/%2A/istio_ai_retry_per_second"
  },
  "items": []
}

This is the rule,

 10   - seriesQuery: 'istio_ai_retry_count{destination_workload="cpuporncheck-default-k8s",reporter="destination"}'
 11     resources:
 12       overrides:
 13         destination_workload: {resource: "pod"}
 14         destination_workload_namespace: {resource: "namespace"}
 15     name:
 16       matches: "^(.*)_count"
 17       as: "${1}_per_second"
 18     metricsQuery: "sum(rate(istio_ai_retry_count{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)"

Can anyone help me to check what's wrong with it?

hackergodness
  • 3
  • 4
  • What kind of metrics is this `istio_ai_retry_count`? Also, is this metric accessible from promethus dashboard? – Matt Mar 26 '20 at 09:50
  • Hi HelloWorld, what do you mean about 'kind'，such as deployment, service? I can get metric values through dashboard. The query is `istio_ai_retry_count{destination_workload="cpuporncheck-default-k8s",reporter="destination"}` – hackergodness Mar 27 '20 at 02:57
  • No, I mean what is exposing this metric? [Googling it doesn't help](https://www.google.com/search?q=istio_ai_retry_count). I'd like to replicate your setup. – Matt Mar 27 '20 at 08:23
  • It is collected by mixer of istio. I may find some clues from [here](https://stackoverflow.com/questions/51840970/how-to-use-istios-prometheus-to-configure-kubernetes-hpa/52042477). But I don't konw whether it's useful. – hackergodness Mar 30 '20 at 01:46
  • What's the output of `istio_ai_retry_count{destination_workload="cpuporncheck-default-k8s",reporter="destination"}` – Matt Mar 30 '20 at 14:33
  • Hi HelloWorld, this problem has been solved. I changed 'pod' to 'deployment' in the rules. Also the access api should like ·/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace_name}/deployment/{deployment_name}/{metric}` – hackergodness Apr 07 '20 at 02:32
  • Would you mind posting it as the answer? – Matt Apr 07 '20 at 06:40
  • No, that's all right. – hackergodness Apr 15 '20 at 03:28

1 Answers1

0

OP confirmed that the problem was with rules.

Changeing pod to deployment and api access path to sth like:

 /apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace_name}/deployment/{deployment_name}/{metric}

resolved the issue.

Matt
  • 7,419
  • 1
  • 11
  • 22