1

I'm new to Grafana and ElasticSearch. We have data in ES with log records of a processing flow with multiple microservices. There are several types of those microservices. For simplicity, let's assume that there are two type of microservices 'task1' and 'task2' and each processing flow has a unique traceId value. The data is something like the following:

{traceId: 'job1', service: 'task1', eventTime: '2021-02-19 12:01:00'}
{traceId: 'job2', service: 'task1', eventTime: '2021-02-19 12:02:00'}
{traceId: 'job1', service: 'task2', eventTime: '2021-02-19 12:05:00'}
{traceId: 'job2', service: 'task2', eventTime: '2021-02-19 12:10:00'}

I would like to add a grafana panel to our dashboard that will have the elapsed time (the time difference between 'task1' and 'task2') for each processing flow ('job1', 'job2'...). I search for similar issue and found the discussion in how to create query which can calculate time difference?, but I'm not sure if the solution described there is suitable for my problem and I couldn't find a way to add the aggregate filter described there. I guess it is way above my current knowledge. Any help would be appreciate

egur
  • 21
  • 3

2 Answers2

1

I've received some tips offline and the following looks like a good approach (but definitely not straight forward IMHO):

Using Raw Data in the query tab and in the transformation tab selecting the "Group by" translation and there using a "group by" translation on the desired field ("traceId" in the above example) and "Calculate" translation for the time field ("eventTime" in the above example) with "Range" as the stats field. This will group the data by the traceId field and display in the table the difference (Range) between the higher value and the lower value of the records with the same traceId.

I'll try to "play" with it and see if I can get the desired graph, anyway - it may help other with the same or similar issues so I post this as at least a direction to solve this issue.

egur
  • 21
  • 3
0

Another approach I recently tried is using the bucket script combined with the Group By options. Something like:

metric(1) - Min eventTime
metric(2) - Max eventTime
metric(3) - Bucket Script with the following options:
                      var1 Min eventTime
                      var2 Max eventTime
                Script:  params.var2 - params.var1
Group By:     traceId 
and then by:  service

Then I used a transformation Organized fields to remove the Min and MaX fields and rename the "Bucket Script" to something like "Time Difference"

egur
  • 21
  • 3