0

I have prometheus data endpoint as - http_request_duration_seconds{method="GET",status="200",uri="XYZ"}1468.0.

How can I create a panel in Grafana for count of http requests with a duration less than or equal to "0.7" seconds?

I have tried few combinations like increase(http_request_duration_seconds[10m]) function, and rate function but getting no data as output. Any help would be appreciated.

Max
  • 1
  • 1
  • 3

1 Answers1

1

You can't do it like that. If you have the plain metric http_request_duration_seconds then it is a summary and you could do something like this:

http_request_duration_seconds{quantile="0.95"}

But in your case this doesn't help because you don't know which quantile represents which exact duration.

Usually you just have http_request_duration_seconds_sum and http_request_duration_seconds_count. These metrics get increased which each request. If you want to calculate the average over a given time you do something like:

 rate(http_request_duration_seconds_sum[5m])
 /
 rate(http_request_duration_seconds_count[5m])

as mentioned here: https://prometheus.io/docs/practices/histograms/

But if you want the exact counts and display a histogram you need to expose buckets, also explained in the above mentioned link and here: https://prometheus.io/docs/concepts/metric_types/#histogram

So you have to use a bucket like so (all requests with a duration up to 0.7s in the last minute):

increase(http_request_duration_seconds_bucket{le="0.7"}[1m])

If you use micrometer and spring this could also help:

Perry45
  • 66
  • 7
  • Thank @Perry45. I want to create a panel for percentage of count of http requests with a duration less than or equal to "0.7" seconds divide by all the http request. I have only two prometheus end points - 1. http_request_duration_seconds_sum{method="GET",status="200",uri="XYZ"}; 2.http_request_duration_seconds_count{method="GET",status="200",uri="XYZ"}; I don't have any prometheus end point that provide bucket information as you mentioned (increase(http_request_duration_seconds_bucket{le="0.7"}[1m])). Is there any way to implement it without the bucket metrics present – Max Apr 07 '22 at 04:07
  • How to expose buckets ? – Max Apr 07 '22 at 06:20
  • I know no other way. Well you can build a custom metric and count yourself but probably that would be a hassle. You have to provide more information, which kind of application do you have (the second link mentioned links to solutions for libs)? Which kind of of monitoring lib do you use? If micrometer: https://micrometer.io/docs/concepts#_histograms_and_percentiles. If you use spring-boot (with micrometer) you can just do metrics.distribution.percentiles-histogram.http.server.requests = true or something – Perry45 Apr 07 '22 at 11:04
  • I am using application spring boot (with micrometer). In build.gradle I am adding the dependency for actuator and prometheus. Can you please specify more on where I can enable the metrics.distribution.percentiles-histogram.http.server.requests = true parameter in my spring boot application ?@Perry45 – Max Apr 07 '22 at 12:37
  • Actually it's management.metrics.distribution.percentiles-histogram.http.server.requests See more infos here https://spring.io/blog/2018/03/16/micrometer-spring-boot-2-s-new-application-metrics-collector or here https://docs.spring.io/spring-boot/docs/2.1.9.RELEASE/reference/html/production-ready-metrics.html you have to set it in your spring properties or yaml. And you should set the sla property to 700ms. Just search in the mentioned links for this property or search it in google, you will find examples and explanations. – Perry45 Apr 07 '22 at 15:53
  • I enabled histogram in my application but I am getting very strange bucketizing like - http_request_duration_seconds_bucket{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/get/details",le="0.20132659"} 119.0; why I am getting so granular value of le ? @Perry45 – Max Apr 12 '22 at 12:32
  • @Vinay That's normal. Look: https://github.com/micrometer-metrics/micrometer/issues/1947 . You can control the min and max values with management.metrics.distribution.minimum-expected-value and management.metrics.distribution.maximum-expected-value like mentioned here: https://docs.spring.io/spring-boot/docs/2.1.9.RELEASE/reference/html/production-ready-metrics.html#_per_meter_properties . And you can add additional buckets for example for 0.7 seconds with management.metrics.distribution.sla. But often you don't care for the buckets itself and aggregate them with histogram_quantile. – Perry45 Apr 12 '22 at 20:04
  • For exmaple: # http SLA histogram buckets management.metrics.distribution.sla.http.server.requests=100ms,150ms,250ms,500ms,1s This would set buckets for http.server.requests up to 100ms, 150ms etc. If sla don't work try slo, because sla is deprecated, it depends on version. See: https://docs.spring.io/spring-boot/docs/2.7.x/reference/html/actuator.html#actuator.metrics.customizing.per-meter-properties – Perry45 Apr 12 '22 at 20:14