33

Is there a way to group all metrics of an app by metric names? A portion from a query listing all metrics for an app (i.e. {app="bar"}) :

ch_qos_logback_core_Appender_all_total{affiliation="foo",app="bar", instance="baz-3-dasp",job="kubernetes-service-endpoints",kubernetes_name="bar",kubernetes_namespace="foobarz",kubernetes_node="mypaas-dev-node3.fud.com",updatedBy="janedoe"}   44
ch_qos_logback_core_Appender_debug_total{affiliation="foo",app="bar", instance="baz-3-dasp",job="kubernetes-service-endpoints",kubernetes_name="bar",kubernetes_namespace="foobarz",kubernetes_node="mypaas-dev-node23.fud.com",updatedBy="deppba"} 32

I have also tried to use wildcard in the metric name, prometheus is complaining about that. Looking at the metrics, I can see that some of them have dynamic names, most probably delivered by dropwizard metrics. What I ultimately want is a list of all available metrics.

naimdjon
  • 3,162
  • 1
  • 20
  • 41

3 Answers3

51

The following query lists all available metrics:

sum by(__name__)({app="bar"})

Where bar is the application name, as you can see in the log entries posted in the question.

naimdjon
  • 3,162
  • 1
  • 20
  • 41
21

{__name__=~".+"} will return all non-stale time series, however this is an expensive query and should be generally avoided.

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • Yes, but it is only to figure out which metrics are there. Tried the query. Error executing query: Gateway Time-out. – naimdjon Mar 07 '18 at 12:06
  • Error executing query: invalid parameter 'query': 1:1: parse error: metric name must not be set twice: "metric_name" or ".+" – Boris Ivanov Jun 18 '21 at 14:38
19

Using directly 

{__name__=~".+"}

will return success but nothing else (too big).

or 

{__name__=~".*"}

while will give us error instead as expected:

parse error at char 17: vector selector must contain at least one non-empty matcher

So my trick here is combining the solutions of brian-brazil and naimdjon using things like

sum({__name__=~"c.*|e.*|n.*|p.*|r.*|k.*|z.*|r.*"}) by (__name__)

  1. since I know the possible prefixes, so I add them to query to make sure at lest it will return something; and further,

  2. to avoid useless fields returned (stressing out the Prometheus), I will only require __name__ in this way by(__name__), all the names of the metrics I need returned as expected.

Actually there is an API to get all the available metric names as:

/api/v1/label/__name__/values
Hearen
  • 7,420
  • 4
  • 53
  • 63