Query Prometheus data using Kusto Query Language

Question

I'm struggling trying to set up some Grafana dashboards using Prometheus data stored in Azure Monitor.

So I have:

An ASP.NET Core application running on AKS that exports Prometheus data via a /metrics endpoint
I've configured Azure Monitor to scrape the /metrics endpoint
And now I'm trying to show a couple of charts in Grafana

But I'm struggling with the Kusto Query Language.

This is an example output of the /metrics endpoint

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1613573002.92
# HELP http_requests_in_progress The number of requests currently in progress in the ASP.NET Core pipeline. One series without controller/action label values counts all in-progress requests, with separate series existing for each controller-action pair.
# TYPE http_requests_in_progress gauge
http_requests_in_progress{method="GET",controller="",action=""} 0
http_requests_in_progress{method="GET",controller="MyController",action="GetSecrets"} 0
# HELP dotnet_collection_count_total GC collection count
# TYPE dotnet_collection_count_total counter
dotnet_collection_count_total{generation="0"} 66
dotnet_collection_count_total{generation="2"} 1
dotnet_collection_count_total{generation="1"} 7
# HELP process_private_memory_bytes Process private memory size
# TYPE process_private_memory_bytes gauge
process_private_memory_bytes 182657024
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 151.03
# HELP http_requests_received_total Provides the count of HTTP requests that have been processed by the ASP.NET Core pipeline.
# TYPE http_requests_received_total counter
http_requests_received_total{code="404",method="GET",controller="",action=""} 1
http_requests_received_total{code="200",method="GET",controller="MyController",action="GetSecrets"} 4
http_requests_received_total{code="200",method="GET",controller="",action=""} 23
http_requests_received_total{code="500",method="GET",controller="MyController",action="GetSecrets"} 2
# HELP process_num_threads Total number of threads
# TYPE process_num_threads gauge
process_num_threads 14
# HELP dotnet_total_memory_bytes Total known allocated memory
# TYPE dotnet_total_memory_bytes gauge
dotnet_total_memory_bytes 5438264
# HELP process_working_set_bytes Process working set
# TYPE process_working_set_bytes gauge
process_working_set_bytes 111087616
# HELP http_request_duration_seconds The duration of HTTP requests processed by an ASP.NET Core application.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_sum{code="404",method="GET",controller="",action=""} 0.0047621
http_request_duration_seconds_count{code="404",method="GET",controller="",action=""} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.001"} 0
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.002"} 0
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.004"} 0
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.008"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.016"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.032"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.064"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.128"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.256"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="0.512"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="1.024"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="2.048"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="4.096"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="8.192"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="16.384"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="32.768"} 1
http_request_duration_seconds_bucket{code="404",method="GET",controller="",action="",le="+Inf"} 1
http_request_duration_seconds_sum{code="200",method="GET",controller="MyController",action="GetSecrets"} 34.2080351
http_request_duration_seconds_count{code="200",method="GET",controller="MyController",action="GetSecrets"} 4
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.001"} 0
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.002"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.004"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.008"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.016"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.032"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.064"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.128"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.256"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="0.512"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="1.024"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="2.048"} 1
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="4.096"} 3
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="8.192"} 3
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="16.384"} 3
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="32.768"} 4
http_request_duration_seconds_bucket{code="200",method="GET",controller="MyController",action="GetSecrets",le="+Inf"} 4
http_request_duration_seconds_sum{code="200",method="GET",controller="",action=""} 0.0218133
http_request_duration_seconds_count{code="200",method="GET",controller="",action=""} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.001"} 21
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.002"} 21
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.004"} 21
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.008"} 22
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.016"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.032"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.064"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.128"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.256"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="0.512"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="1.024"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="2.048"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="4.096"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="8.192"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="16.384"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="32.768"} 23
http_request_duration_seconds_bucket{code="200",method="GET",controller="",action="",le="+Inf"} 23
http_request_duration_seconds_sum{code="500",method="GET",controller="MyController",action="GetSecrets"} 620.8115536
http_request_duration_seconds_count{code="500",method="GET",controller="MyController",action="GetSecrets"} 2
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.001"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.002"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.004"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.008"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.016"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.032"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.064"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.128"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.256"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="0.512"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="1.024"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="2.048"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="4.096"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="8.192"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="16.384"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="32.768"} 0
http_request_duration_seconds_bucket{code="500",method="GET",controller="MyController",action="GetSecrets",le="+Inf"} 2
# HELP process_open_handles Number of open handles
# TYPE process_open_handles gauge
process_open_handles 196
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 3918102528

And the data that ends ups in Azure Monitor has (among other things fields like Name, Val, Tags, etc). For example

I'm trying to set up a couple of different charts:

Total number of requests per interval. For example average number of requests per 5 minute, during the past 24 hours
Total number of 2xx requests per interval.
Same for memory/CPU

I guess this would be dirt simple, but I just can't get it right...

I've got something like this:

InsightsMetrics 
| where $__timeFilter(TimeGenerated)
| where Namespace == "prometheus"
| where Name == "http_requests_received_total"
| extend dimensions=parse_json(Tags)
| where dimensions.app == "myappname"
| summarize count() by Val, bin(TimeGenerated, $__interval) // Don't know what to put here...
| order by TimeGenerated asc

I have just found a solution (i'm testing it right now). If you want I can provide more details tomorrow if you still need it. — Adam Łepkowski, Nov 08 '21 at 16:14
@AdamŁepkowski I do not need it at the moment, but would still be interested in the details! You never know :) — Joel, Nov 08 '21 at 18:13
we decided to not store Prometheus data in Log Analytics it's completely immature. There are too many problems with this solution. I can make queries on counter data types but other types are tricky… below you can find a couple of reasons why we didn’t decide to use it: — Adam Łepkowski, Nov 09 '21 at 19:40
1. The agent responsible for storing Prometheus data in Azure Log Analytics store them in InsightsMetrics.Tags field. It may appear that if a metric label name has the value of the pod label then the metric label value is overridden by the value of the pod label. https://techcommunity.microsoft.com/t5/azure-monitor/prometheus-metric-label-value-is-overridden-by-pod-label/m-p/2942278 — Adam Łepkowski, Nov 09 '21 at 19:41
2. To query counter metric types you need to use in most cases two kusto functions series_metric_fl() and series_rate_fl(). There is a bug in the second function and it appears when you have more than one pod (because then you have two metrics in the same time). Additionally, those functions need to have the correct labels field because aggregation is done based on the labels so you need to extract labels from InsightsMetrics.Tags field. — Adam Łepkowski, Nov 09 '21 at 19:41
3. If you would like to work with histogram metric type then I can just say good luck. In Prometheus, there is a function histogram_quantile() but it’s not supported https://github.com/microsoft/ApplicationInsights-dotnet/issues/1226. — Adam Łepkowski, Nov 09 '21 at 19:41
4. It’s hard to build queries that can work with grapahan in grapahan you can specify variables that have the value “ALL” and you can specify that one or many values can be chosen. Now imagine how to build such a query in kusto where there are no conditional filtering (you can use conditional aggregation but it’s tricky and doesn’t work in each scenario). I finished with such things: datatable (a:string, b:string) ['1', '2','3', '4','5', '6'] | where a == "*" or 1==1 VS datatable (a:string, b:string) ['1', '2','3', '4','5', '6'] | where a == $service — Adam Łepkowski, Nov 09 '21 at 19:41
The key is to understand types of metric data in prometheus there are a couple of blogs describing how to use query them but 80% present wrong information... i.e. you don't calculate counter, you need to check between two samples whether counter increase and then you know that even appeared and how many times.. — Adam Łepkowski, Nov 09 '21 at 19:43

Query Prometheus data using Kusto Query Language

0 Answers0