5

I have deployed the Istio Bookinfo application in a Kubernetes cluster. Following the documentation, I'm trying to measure the average request duration using the following query:

rate(istio_request_duration_milliseconds_sum[1m]) / rate(istio_request_duration_milliseconds_count[1m])

This query returns me "doubled" results i.e. for each request I get two results, with different Value but same source and destination, one labelled as reporter="source" and the other as reporter="destination". I was not able to find any clarification about it and I have not clear how these measures work. Why I have two values returned?

Jackie
  • 53
  • 1
  • 3

1 Answers1

5

The metrics are indeed reported from two sources, once from the request sender and another time from the request receiver. This is a bit redundant, but in some situations the values differ and especially for requests duration: from request sender (reporter=source), the duration stands for the whole request latency (server processing time + network roundtrip) whereas from the request receiver (reporter=destination), it stands mainly for server processing time.

In a few special situations there is no redundancy and metrics are only reported once (be it from source or from destination): e.g. when istio features such as fault injection or traffic mirroring come into play.

As a result, in Prometheus you should always filter all your queries with either {reporter="source"} or {reporter="destination"}.

Joel
  • 2,374
  • 17
  • 26