Is there any existing grok{} pattern for date format YYYY/MM/DD HH:mm:ss?

Question

I was checking the nginx error logs at our server and found that they start with date formatted as:

2015/08/30 05:55:20

i.e. YYYY/MM/DD HH:mm:ss. I was trying to find an existing grok date pattern which might help me in parsing this quickly but sadly could not find any such date format. Eventually, I had to write the pattern as:

%{YEAR}/%{MONTHNUM}/%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

I am still hoping if there is a shorter pattern for the same ?

Does the pattern _actually_ have to contain `[T ]` and `%{ISO8601_TIMEZONE}?`? Looking at your single example they're unnecessary. Also, why make the seconds optional? — Magnus Bäck, Sep 06 '15 at 19:59
Agree with you. These are not necessary. But is there any pattern available that can parse this date format as it is? — Mandeep Singh, Sep 07 '15 at 08:56

score 10 · Accepted Answer · answered Sep 07 '15 at 08:13

10

No. You find the included patterns on github. The comment to datestamp seems to fit to your YYYY/MM/DD, but DATE_US and DATE_EU are different.

I suggest overload the DATE pattern using grok option patterns_dir and go with DATESTAMP.

DATE_YMD %{YEAR}/%{MONTHNUM}/%{MONTHDAY}
DATE %{DATE_US}|%{DATE_EU}|%{DATE_YMD}

or just add your pattern into a patterns-file and use grok's patterns_dir option.

answered Sep 07 '15 at 08:13

dtrv

693
1
6
14

But how would that matched pattern be parsed into a proper date type? I think the other answer should be merged here. – Vanuan Jul 27 '16 at 13:14

score 6 · Answer 2 · answered Jul 27 '16 at 13:44

Successful timestamp capture strategy comprised of 3 things

Precision and timezone in the original log. Change your nginx timestamp log format.

Use $msec to capture milliseconds. Otherwise you wouldn't be able to sort it precisely.

log_format custom '[$msec] [$remote_addr] [$remote_user] '
                  '"$request" $status '
                  '"$http_referer" "$http_user_agent"';

Raw timestamp. Use greedy matching to capture raw data into a field.

Use GREEDYDATA:

grok {
  match => { "message" => "\[%{GREEDYDATA:raw_timestamp}\] %{GREEDYDATA:message}" }
  overwrite => [ "message" ]
}

Parsed timestamp. Use date filter to parse raw timestamp.

reference

date {
  match => [ "timestamp", "yyyy/MM/dd HH:mm:ss.S z" ]
  target => "@timestamp"
}

Thiago Falcao · Answer 3 · 2019-04-11T14:50:02.413

6

To match 2015/08/30 05:55:20, use:

%{DATESTAMP:mytimestamp}

Tested on Logstash 6.5

Source: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

edited Apr 11 '19 at 14:50

answered Dec 20 '18 at 17:05

Thiago Falcao

4,463
39
34

score 1 · Answer 4 · edited Jan 26 '17 at 06:58

1

You can also simply include the joda.time pattern which is simple and short.

date {
  match => [ "timestamp", "yyyy/MM/dd HH:mm:ss" ]
  target => "@timestamp"
}

Helpful link for reference: https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html

edited Jan 26 '17 at 06:58

m0nhawk

22,980
9
45
73

answered May 27 '16 at 16:30

Philip

503
7
8

Is there any existing grok{} pattern for date format YYYY/MM/DD HH:mm:ss?

4 Answers4