0

Im trying to store old log files in elastic and im using logstash for that.

time stamps in the logs are of the following format:

13 AUG 2015 | 07:04:35 | .......

1st problem was the fact the month is in upper case so i copied the "MONTH" pattern as it appears in the grok-patterns and upper-cased all of it:

original MONTH:

MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b

my upper-case "MONTHCAP":

MONTHCAP \b(?:JAN(?:UARY)?|FEB(?:RUARY)?|MAR(?:CH)?|APR(?:IL)?|MAY|JUN(?:E)?|JUL(?:Y)?|AUG(?:UST)?|SEP(?:TEMBER)?|OCT(?:OBER)?|NOV(?:EMBER)?|DEC(?:EMBER)?)\b

next thing i try to do is use the date filter so elastic uses the timestamp from the log as @timestamp filed and NOT the time of the line being stored in elastic:

date{
                match => ["MONTHDAY","dd","MONTHCAP","MMM","YEAR","yyyy","TIME","HH:mm:ss"]
    }

The problem is that i get the following error trying to store the data:

Error: Cannot register filter date plugin. The error reported is:
Illegal pattern component: O for pattern 'MONTHCAP'

additional information: this is the grok filter i use to parse the log lines:

%{MONTHDAY} %{MONTHCAP} %{YEAR} \| %{TIME} \|

any idea why i keep getting this error when thr pattern i use is 'MMM' ? THanks!

Eitan Vesely
  • 125
  • 3
  • 16

1 Answers1

0

In the documentation the match is only for one field :

match => [ "logdate", "MMM dd YYY HH:mm:ss",
      "MMM  d YYY HH:mm:ss", "ISO8601" ]

It takes the first solution which matchs with logdate field.

You need to construct a field which is your date and after you can do your date filter.

 match => [logdate, dd MMM yyyy HH:mm:ss]
mherbert
  • 515
  • 3
  • 12
  • Thanks, i get it. But how do i put all the separate time stamp fields into one ? – Eitan Vesely Aug 18 '15 at 14:47
  • filter { mutate { add_field => { "foo_%{somefield}" => "Hello world, from %{host}" } } } – mherbert Aug 18 '15 at 15:24
  • THanks. now that i got it working and i added all fields to create one as you suggested 'add_field => { "logTimeStamp" => "%{day} %{month} %{year} %{time}" }' but for some reason the date filter still doesn't work :( date{ match =>["logTimeStamp","dd MMM YYYY HH:mm:ss" ] }. could it be the fact that the month are is upper case letters?? i.e JAN,FEB etc. and the MMM represents Jan,Feb,... etc? – Eitan Vesely Aug 25 '15 at 19:27