Logstash - Trying to create an elapsed filter

Question

I'm trying to create an elapsed filter but the elapsed fields don't appear. This is the input:

statement => "SELECT TRANSACTION_ID, COMMUNICATION_ID, 
    BROKER_NAME, IS_NAME, SERVICE_NAME, OPERATION_NAME, OPERATION_VERSION, MESSAGE_TYPE, APPROACH, CLIENT_ID, 
    APPLICATION_ID, EXT_SESSION_ID, EXT_TRANSACTION_ID, EXT_ORIGIN, LANG_CODE, EXT_HOST, APPLICATION, CHANNEL, 
    NUM_RETRIES, STATUS_CODE, STATUS_MSG, DATE_CREATED, 
    DESTINATION_HOST, OPERATION_ID 
    FROM IIB_OPER.COMMUNICATION_LOG 
    WHERE DATE_CREATED > '2018-07-20'"

And this is the filter:

filter {
    if [message_type] == "Req" {
        mutate {
            add_tag => [ "taskStarted" ]
        }
    }
    if [message_type] == "Res" {
        mutate {
            add_tag => [ "taskTerminated" ]
        }
    }   
    elapsed {
        unique_id_field => "operation_id"
        start_tag => "taskStarted"
        end_tag => "taskTerminated"
        timeout => 20000
        new_event_on_match => true
    }
}

In Kibana, in the index patterns, the fields appear but when i get the logstash to work the elapsed fields don't appear.

Any idea why?

Cheers,

score 0 · Accepted Answer · answered Jul 30 '18 at 10:58

Answering my own question... The problem is i'm trying to transform a column that was already in the JSON to import to Elastic, so making another temporary date it works.

date {
    match => [ "temp_date", "yyyy-MM-dd HH:mm:ss,SSS"]
}
if [message_type] == "Req" {
    mutate {
        add_tag => [ "taskStarted" ]
    }
}
if [message_type] == "Res" {
    mutate {
        add_tag => [ "taskTerminated" ]
    }
}   
elapsed {
    unique_id_field => "operation_id"
    start_tag => "taskStarted"
    end_tag => "taskTerminated"
    timeout => 30
}

Another point... very important... the timeout is in secs, no in milis.

Cheers,

Logstash - Trying to create an elapsed filter

1 Answers1