I'll first try to describe the non-technical problem:
There are many services, each service can have multiple instances and each of those instances is scraped for data and that data is stored in influxdb. Now the point in time where that data is scraped from each instance of the service is (obviously) not exactly the same point in time.
What i like to query (or display) is the maximum value for each service, over all instances. And i did not find a way to "quantify" time points. For examle to always move the value to the next full minute or something so that timescales get comparable.
Not the technical problem is that the reported values are all totals so to get a sense of change in that values i need difference oder derivative but these functions are in my case often aplied to one value of instance 1 and one value of instance 2 which reflects the difference between the instances and not between two points in time.
heres what i tried so far but i gives me almost flatlines since the instances report pretty much the same value but at alternating points in time.
from(bucket: "dpl4")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r._measurement == "prometheus")
|> filter(fn: (r) => r._field == "http_server_requests_seconds_sum")
|> group(columns: ["service"], mode: "by")
|> aggregateWindow(every: 1m, fn: max)
|> derivative(unit: 1m, nonNegative: true)
I hope i was able to describe the problem.
