I have an (.NET Core) app that makes a specific request to third party that we need to monitor quite closely for performance.
The app uses prometheus-net for measuring operation duration - see here.
The code that invokes that invokes the third party service and measures the operation duration is as follows. Note: I've created my own timer to track progress in tens of milliseconds (as we need this level of precision).
private static readonly Histogram RequestDuration = Metrics
.CreateHistogram("request_duration", "Histogram of request durations",
new HistogramConfiguration { Buckets = Histogram.LinearBuckets(1, 1, 100) });
using var timer = RequestDuration.InTensOfMilliseconds();
using var tokenSource = new CancellationTokenSource(TimeSpan.FromMilliseconds(_settings.RequestTimeoutInMilliseconds));
response = await httpClient.PostAsJsonAsync("api/v1/transaction", transactionRequest, defaultJsonSerializerOptions, tokenSource.Token);
If I look at the (prometheus) metrics endpoint, I can see the linear buckets which each contain the total number of requests that came in at less than or equal to the specified number of 'tens of milliseconds' e.g.
request_duration_count 8
request_duration_bucket{le="1"} 0
request_duration_bucket{le="2"} 0
request_duration_bucket{le="3"} 4
request_duration_bucket{le="4"} 7
request_duration_bucket{le="5"} 7
request_duration_bucket{le="6"} 7
request_duration_bucket{le="7"} 7
request_duration_bucket{le="8"} 7
request_duration_bucket{le="9"} 7
request_duration_bucket{le="10"} 7
...
request_duration_bucket{le="99"} 7
request_duration_bucket{le="100"} 7
request_duration_bucket{le="+Inf"} 8
We're intending to send our telemetry data (logs, metrics, etc) to Elastic Cloud. Metrics like the request duration above will be scraped periodically (e.g. every X seconds/minutes) from the metrics endpoint of the app.
How would I go about creating a visualization in Kibana to show a histogram of aggregated request durations e.g. something like this (from Google images):
