Logstash - Use current date as timestamp date

Question

I would like to use the current day as timestamp (date) as this information isn't available in our logfiles. Example -> main_core.log:

04:00:19.675 [ActiveMQ Task-9] INFO  a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345

And i would like to split it and use the current day as date and the timestamp out of the logfile.

Is that even possible?

Thanks and many regards

score 2 · Accepted Answer · answered Jul 12 '18 at 20:12

You can add a field with the part of the timestamp that is missing in your log and then concatenate with a variable that contains the hour and use it as your @timestamp field.

The filter below does something like this:

filter {
    grok {
        break_on_match => false
        match => ["message","%{TIME:hour} %{GREEDYDATA:msg}"]
        tag_on_failure => [ "_grokparsefailure"]
        add_field => { "time" => "%{+YYYY-MM-dd}"}
        add_field => { "timestamp" => "%{time} %{hour}" }
    }
    date {
        target => "@timestamp"
        match => ["timestamp", "YYYY-MM-dd HH:mm:ss.SSS"]
    }
}

First it will match your message with a grok pattern that will extract the hour and save it in a field name hour, and the rest will be saved in a field name msg, but you can parse the rest if you want.

Then it will add a field name time with the pattern YYYY-MM-dd, for example 2018-07-12.

After that, it will create a field named timestamp the field time with the field hour, which will result in 2018-07-12 4:00:19.675

The date filter is used to use your generated timestamp as the default timestamp field in elastic, which is @timestamp.

A logstash output for this filter is something like this:

{
 "@timestamp":"2018-07-12T04:00:19.675Z",
 "message":"04:00:19.675 [ActiveMQ Task-9] INFO a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345",
 "timestamp":"2018-07-12 04:00:19.675",
 "msg":"[ActiveMQ Task-9] INFO  a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345",
 "time":"2018-07-12",
 "@version":"1",
 "hour":"04:00:19.675",
 "host":"logstash-hostname"
}

Logstash - Use current date as timestamp date

1 Answers1