Filter vector based new label introduced by `label_replace`

Question

In Prometheus I'm trying to merge multiple http request lines into groups using label_replace.

http_requests_total{account_id="124",handler="AAAAAController"...}
http_requests_total{account_id="125",handler="BBBBBController"...}
http_requests_total{account_id="126",handler="CCCCCController"...}
http_requests_total{account_id="123",handler="XXXXXController"...}

The query I wrote is :

label_replace(http_requests_total, "class", "$1", "handler", "([a-zA-Z0-9]+)Controller.*") .

This works correctly and adds the class label to vector :"AAAA","BBBB" etc. At this point I would like to remove certain classes such as empty and BBBB.

How can I further filter the vector using {class~="BBBBB"} :

label_replace(http_requests_total, "class", "", "handler", "([a-zA-Z0-9]+)Controller.*"){class~="BBBBB"}

Prometheus shows an error when I try to do so.

Answer 1

The filtering for this particular question can be done via the inner series selector before applying label_replace. For example, the following query returns only series with class=~"BBBBB":

label_replace(
  http_requests_total{handler=~"BBBBBController.*"},
  "class",
  "$1",
  "handler",
  "([a-zA-Z0-9]+)Controller.*"
)

P.S. While Prometheus doesn't provide the ability to post-filter time series by label values after they are selected by series selector, this can be done in VictoriaMetrics - Prometheus-like system I work on. See label_match and label_mismatch functions.

Answer 2

I've noticed that there is a grafana tag under this question, so if you don't mind to use grafana to achive that goal, then there is another way.

By using Grafana... 7.x+ IIRC, you can use Transform to reprocess the raw result set.

For instance, I have a Query result like this:

Add a Filter data by values plugin. You can set Match any in it, and add a regex condition. Set field Class matches BBBBB or anything you want.

Answer 3

I think the record rules feature might be helpful to you. Following is the steps for your reference

1. create record rule. Prometheus allow us to set 2 type rules, one is alert rule another one is record rule. Record rule will create new metrics by the setting that you provide on Prometheus server setting. Here is the sample according your description.

groups:
- name: http_requests_total_new_merics
  rules:
  - record: http_requests_total_new
    expr: label_replace(http_requests_total, "class", "", "handler", "([a-zA-Z0-9]+)Controller.*")

then Prometheus will create new metrics named http_requests_total_new

2. Using PromQL to query according your needs, e.g.

• http_requests_total_new{class!="BBBBB"}

Answer 4

solution:

You can achieve it via joining two metrics, for example I need to group pods by nodes filtered by the name of the deployment that I pass as grafana variable

count by (node)
(
  label_replace(kube_pod_info{namespace="prod"}, "deployment", "$1", "pod", "(.*)-(.*)-(.*)$") 
  * on (deployment) group_left(job) 
       kube_deployment_labels{namespace="prod", deployment=~"prod-celery-beat-v1|prod-api-v1|prod-backend-v1"}
)

Answer 5

I had the same problems and this is my solution:

label_repalce({__name__=~"foo($lable)"}, 'lable', ...)

The label is which you will generate, you can set it on Grafana. This method works for me.