0

I've been reviewing different ways to aggregate log messages together that have a start event but no end event. Been struggling with the logstash aggregate filter plugin not sorting correctly and was looking at retrofitting an old entity-centric model for a previous version of elasticsearch Entity-Centric Indexing - Mark Harwood | Elastic Videos when I realized elasticsearch 7.13 transforms introduce the concept of 'latest' which negates my need for a bunch of external scripts (hopefully) to do this.

I am looking at the "Getting Web Session Details by using Scripted Metric Aggregation" sample painless script https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-painless-examples.html#painless-web-session which produces session details, including session duration. Because the logs do not have an end-time, I need to make use of a timeout interval, something like a 30 minute window for aggregating message events based on my group by.

Is this possible to do within the transform by adjusting that script and could anyone help?

jjammin92
  • 25
  • 4
  • 1
    Here are two examples that you can get some inspiration from: https://stackoverflow.com/a/35204605/4604579 + https://stackoverflow.com/a/37359000/4604579 – Val Jul 27 '21 at 15:13
  • Thank you for the provided information. I was previously utilizing the aggregate filter plugin for logstash as you linked to. I was running into issues with this, and given its written in ruby, I'm unsure of the functions that could work in painless for setting any form of timeout. – jjammin92 Jul 27 '21 at 16:13
  • ruby runs in Logstash, painless runs in ES, those are two completely separate processes, I'm not sure how those could get in conflict. Feel free to shed some more light on what you're trying to achieve and maybe provide the Logstash configuration you have now – Val Jul 27 '21 at 16:20
  • I'm not trying to utilize logstash. I'm trying to utilize a transform, and adjust the provided painless script in the example to set a timeout interval. – jjammin92 Jul 27 '21 at 17:22

0 Answers0