4

I would like compare the following two time series which are different from each other. All I want to do is divide the result of the Second time series by the result of the First time series and generate an alert if the value is less than 1. The query that I am using is below the time series that I have mentioned below. How to achieve my goal?

First time series:

container_memory_usage_bytes{container_name="documentation"}

RESULT:

container_memory_usage_bytes{
   beta_kubernetes_io_arch="amd64",
   beta_kubernetes_io_fluentd_ds_ready="true",
   beta_kubernetes_io_instance_type="n1-highmem-8",
   beta_kubernetes_io_os="linux",
   cloud_google_com_gke_nodepool="pool-1",
   container_name="documentation",
   failure_domain_beta_kubernetes_io_region="europe-west1",
   failure_domain_beta_kubernetes_io_zone="europe-west1-b",
   id="/kubepods/burstable/podb79239ff-1ee9-11e8-a6c1-42010a840fd9/3ccdaa6597d5bd306d228e37d9d4c65a9681dba2e894aa7b7ed4502ec54e5619",
   image="gcr.io/ingenious-169318/publicapi/documentation@sha256:60a5454bb40ed34f24cbeb9f330e1097191754cf2890eee1ca4f2988188a7705",
   instance="gke-ingenious-pool-1-bbd77706-5rbz",
   job="kubernetes-cadvisor",
   kubernetes_io_hostname="gke-ingenious-pool-1-bbd77706-5rbz",
   name="k8s_documentation_documentation-6c4c4c9f5f-dn2bv_apimanager_b79239ff-1ee9-11e8-a6c1-42010a840fd9_0",
   namespace="apimanager",
   pod_name="documentation-6c4c4c9f5f-dn2bv"
}

Second time series:-

kube_pod_container_resource_requests_memory_bytes{container="documentation"}

RESULT: 

kube_pod_container_resource_requests_memory_bytes{
   container="documentation",
   instance="10.32.0.30:8080",
   job="kubernetes-service-endpoints",
   k8s_app="kube-state-metrics",
   kubernetes_name="kube-state-metrics",
   kubernetes_namespace="kube-system",
   namespace="apimanager",
   node="gke-ingenious-pool-1-bbd77706-pdsz",
   pod="documentation-6c4c4c9f5f-mzd54"
}

PromQL:

kube_pod_container_resource_requests_memory_bytes{container="documentation"} / (container_memory_usage_bytes{namespace="apimanager"})

James Z
  • 12,209
  • 10
  • 24
  • 44
Suhas Chikkanna
  • 1,292
  • 5
  • 20
  • 34

1 Answers1

9

Vector Matching

Prometheus calls this Vector Matching, you can perform arithmetic binary operations (+, -, *, / etc) across different time series if their labels match. If they don't match you can using on or ignoring to chose which labels you should match on.

One-to-one finds a unique pair of entries from each side of the operation. In the default case, that is an operation following the format vector1 vector2. Two entries match if they have the exact same set of labels and corresponding values. The ignoring keyword allows ignoring certain labels when matching, while the on keyword allows reducing the set of considered labels to a provided list

e.g.

kube_pod_container_resource_requests_memory_bytes{container="documentation"}
/ on (container)
container_memory_usage_bytes{namespace="apimanager"}

label_join/label_replace

Looking at your specific example there aren't any labels that you could use on to match with. container_memory_usage_bytes has container_name while kube_pod_container_resource_requests_memory_bytes has container. Prometheus has other query functions called label_join and label_replace which allow you to create custom labels on one of your metrics.

In your case you might want to use label_replace to append _name to kube_pod_container_resource_requests_memory_bytes. Which would then allow you to use / on (container_name) as shown above.

Hopefully these 2 prometheus concepts, vector matching and label_join/replace will help you perform arithmetic operations across disparate time series. NB I'm not familiar enough with your exact use case to know if container_name is the best label to use to achieve your required output or if there is a simpler way of achieving this with Kubernetes and Prometheus.

Richard Paul
  • 276
  • 1
  • 5
  • 15