0

I am using the latest version of logstash(7.6.2). I am trying to calculate the total time spent in system excluding time spent between Out2 and In2 i.e. [(Out3-In1) - (Out2-In2)] for a given ID. Find below my sample data with columns Timestamp, Event, and ID

Sample Data

18/May/2015:02:05:10 +0000 In1 100
18/May/2015:02:05:15 +0000 Out1 100
18/May/2015:02:05:26 +0000 In2 100
18/May/2015:02:05:48 +0000 Out2 100
18/May/2015:15:05:08 +0000 In3 100
18/May/2015:16:05:03 +0000 Out3 100

I am currently using two elapsed functions and 2 aggregate functions as shown below in the config file

Config File

filter{
    grok{
        match => {"message" => "%{HTTPDATE:timestamp} %{WORD:Event} %{NUMBER:ID}"}
        add_tag => ["%{Event}"]
    }
    date{
        match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
    elapsed {
        start_tag => "In1"
        end_tag => "Out3"
        unique_id_field => "ID"
    }
    elapsed {
        start_tag => "In2"
        end_tag => "Out2"
        unique_id_field => "ID"
    }
    if [Event] == "Out2" {
        aggregate {
            task_id => "%{ID}"
            code => "map['time1'] = (event.get('elapsed_time')*1000).to_i"
            map_action => "create"
        }
    }
    if [Event] == "Out3" {
        aggregate {
            task_id => "%{ID}"
            code => "map['time2'] = (event.get('elapsed_time')*1000).to_i - map['time1']; event.set('time1', map['time2'])"
            map_action => "update"
            end_of_task => true
        }
    }
}

I was able to generate elapsed time fields but was unable to calculate difference between then using Agreggate filter. Find below my output

Is there anything I am missing?

Any help is appreciated! Thanks in advance

Nani
  • 260
  • 2
  • 9
  • Are you using only one worker on your pipeline? – leandrojmp Apr 27 '20 at 19:27
  • No. The worker is set to default. I didn't change any settings in pipeline.yml. – Nani Apr 27 '20 at 19:43
  • 1
    You need to set your `pipeline.workers` equals to 1 if you want to use the `aggregate` filter. [aggregate filter documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-description) – leandrojmp Apr 27 '20 at 19:59
  • @leandrojmp I tried adding ```pipeline.workers:1``` in my ```pipelines.yml``` and had re-run the logstash upload. But, the ```_aggregateexception``` persists and additionally, I also received an ```elapsed_end_without_start``` tag. – Nani Apr 28 '20 at 06:05
  • @leandrojmp My bad. I have added ```pipeline.workers:1``` in ```pipelines.yml``` instead of ```logstash.yml```. I have made the correction. It's working now! Thanks a lot! – Nani Apr 28 '20 at 06:47

0 Answers0