Logstash filter timestamp from log message

Question

I have been following the Logstash tutorial and created the following config file for test purposes:

input {
  file {
    path => "C:\Dev\sample.log"
    start_position => beginning
  }
}
filter{
    date {
        match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
    }
}
output {
  elasticsearch { host => localhost
                  index => "test"
                }
  stdout { codec => rubydebug }
}

However, the only fields that are output are "message", "@version", "@timestamp", "host" and "path".

No "logdate" is retrieved. I have searched for a while and I saw people having the same problem given wrong date format, but I checked mine with "Joda-Time" just as Logstash tutorial recommends. Thank you for your help.

score 4 · Accepted Answer · answered May 19 '14 at 16:37

I managed to get the result I was looking for by doing the following:

    input {
  file {
    path => "C:\Dev\sample.log"
    start_position => beginning
  }
}
filter{
    grok {
       match => [ "message", "%{TIMESTAMP_ISO8601:logdate}" ]
    }       
    date {
        match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS"]
    }
}
output {
  elasticsearch { host => localhost
                  index => "test"
                }
  stdout { codec => rubydebug }
}

So, this could be an answer to my question and I hope it can help someone that got stuck too. However I still don't understand why my previous example (in the question) does not give me the (which I think is) the correct output? I would really appreciate if someone could give me some explanation. Many thanks!

score 1 · Answer 2 · answered May 19 '14 at 17:02

First I need to say that I am only new to logstash but my understanding is as follows:

In the date filter, match is a bit of a confusing term to use. It does not match in the regex sense of the word, it parses a string and turns it into a date. It simply looks at a string and says, "the first character is the first digit of the year" etc. What it does not do is find a string of the specified format in another string (the whole message in your case). That is the job of grok.

So to summarise, grok finds a string that looks like a date and the date filter turns a string into a date.

Logstash filter timestamp from log message

2 Answers2