grok not matching timestamp to date '13 Oct 2015 21:30:26'

Question

I've been trying for hours now, but grok simply doesn't want to parse the timestamp correctly:

Message:

Tue, 13 Oct 2015 21:30:26 GMT users_service Three users logged in.

.conf file:

input { stdin { } }

filter {
  grok {
    match => { "message" => "%{DAY:day}, %{MONTHDAY:month_day} %{MONTH:month} %{YEAR:year} %{TIME:time} GMT %{WORD:service} %{GREEDYDATA:message_entry}" }
    add_field => [ "received_at", "%{@timestamp}" ]
    add_field => [ "received_from", "%{host}" ]
  }
  date {
    match => [ "timestamp", "dd MMM yyyy HH:mm:ss" ]
    locale => "en"
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

What I get is:

{
          "message" => "Tue, 13 Oct 2015 21:30:26 GMT users_service Three users logged in.",
         "@version" => "1",
       "@timestamp" => "2015-10-13T23:09:58.738Z",
             "host" => "users_host",
              "day" => "Tue",
        "month_day" => "13",
            "month" => "Oct",
             "year" => "2015",
             "time" => "21:30:26",
          "service" => "users_service",
    "message_entry" => "Three users logged in.",
      "received_at" => "2015-10-13T23:09:58.738Z",
    "received_from" => "users_host"
}

I was expecting to have a timestamp field in there, but there isn't any.

score 0 · Answer 1 · answered Oct 13 '15 at 23:26

0

As you can see, grok{} is making several fields for you. date{} is expecting a single field and a format for that field, but nothing in the config is combining them back into a single 'timestamp' field that you can use.

You have two choices: make a different grok pattern that puts all of the date/time stuff into one field and then pass that (with the correct pattern info) to date{}, or use add_field to make a new field from all the pieces you have.

answered Oct 13 '15 at 23:26

Alain Collins

16,268
2
32
55

How would that `add_field` look exactly? Something like `add_field => [ "timestamp", "%{month_day} %{month} %{year} ... " ]` ? – cgf Oct 13 '15 at 23:31
Sorry, wrong question. I'm interested in the date{} approach. I created the pattern, but how do I pass that with the correct format to date => match ? – cgf Oct 13 '15 at 23:36
Can you update the posting with the current state of affairs? – Alain Collins Oct 14 '15 at 00:07

score 0 · Answer 2 · answered Nov 06 '15 at 05:40

The date filter you have configured expects a "timestamp" field in the record. Currently your record does not contain a field with that name and expected contents.

You can create the needed field in your grok filter using the data you have matched in the message and the add_field setting and then remove the temporary field when the date filter is sucessful:

filter {
  grok {
    match => {
      "message" => "%{DAY:day}, %{MONTHDAY:month_day} %{MONTH:month} %{YEAR:year} %{TIME:time} GMT %{WORD:service} %{GREEDYDATA:message_entry}"
    }
    add_field => {
      "received_at" => "%{@timestamp}"
      "received_from" => "%{host}"
      "timestamp" => "%{month_day} %{month} %{year} %{time}"
    }
  }

  date {
    match => [ "timestamp", "dd MMM yyyy HH:mm:ss" ]
    locale => "en"
    remove_field => [ "timestamp" ]
  }
}

grok not matching timestamp to date '13 Oct 2015 21:30:26'

2 Answers2