1

Receiving a parsing failure with my grok match. I can't seem to find anything that will match my log.

Here is my log:

2016-06-14 14:03:42 1.1.1.1 GET /origin-www.site.com/ScriptResource.axd?d= jEHA4v5Z26oA-nbsKDVsBINPydW0esbNCScJdD-RX5iFGr6qqeyJ69OnKDoJgTsDcnI1&t=5f9d5645 200 26222 0 "http://site/ layouts/CategoryPage.aspx?dsNav=N:10014" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G318HZ Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.95 Mobile Safari/537.36" "cookie"

Here is my grok match. It works fine in the grok debugger.

filter {
  grok {
    match => { 'message' => '%{DATE:date} %{TIME:time} %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:status} %{NUMBER:bytes} %{NUMBER:time_taken} %{QUOTEDSTRING:referrer} %{QUOTEDSTRING:user_agent} %{QUOTEDSTRING:cookie}' }
  }
}

EDIT: I decided to do a screenshot of what my log file looks like as the spaces dont come over when copying and pasting. Those appear to be single spaces when I copy/paste.

enter image description here

Will Barnwell
  • 4,049
  • 21
  • 34
maltman
  • 454
  • 1
  • 7
  • 28

1 Answers1

3

Beside the space in that logline you posted, which I assume won't exist in your logs, your pattern is incorrect on the date parsing. Logstash DATE follows this pattern:

DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
DATE %{DATE_US}|%{DATE_EU}

Which doesn't match your YYYY-MM-dd format. I recommend using a pattern file and defining a custom date format

CUST_DATE %{YEAR}-%{MONTHNUM2}-%{MONTHDAY}

then your pattern can be

%{CUST_DATE:date} %{TIME:time} %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:status} %{NUMBER:bytes} %{NUMBER:time_taken} %{QUOTEDSTRING:referrer} %{QUOTEDSTRING:user_agent} %{QUOTEDSTRING:cookie}

EDIT: You may be able to handle weird whitespace with a gsub, this won't remove whitespace, but will normalize spaces to all be 1 " "

mutate {
    gsub => [
      # replace all whitespace characters or multiple adjacent whitespace characters with one space
      "message", "\s+", " "
    ]
  }
Will Barnwell
  • 4,049
  • 21
  • 34
  • Thank you. I am seeing this now in the log. "Failed to parse mapping [_default_]: Mapp ing definition for [@time] has unsupported parameters: [format : hh:mm:ss]", "c aused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"Mapping definition fo r [@time] has unsupported parameters: [format : hh:mm:ss]"}}}}, – maltman Jun 30 '16 at 13:44
  • the funny thing is, I can't find anywhere I have defined time. I created a custom pattern for time as well and I get the same error. – maltman Jun 30 '16 at 15:08
  • adding template_overwrite => true fixed the issue. – maltman Jun 30 '16 at 15:53
  • sorry got another issue. Looks like my log files add a big space between log items. They are single spaces but I think it is throwing off my pattern match. the debugger doesnt even like these spaces. is there anything that can be done? 2016-06-14 14:03:42 1.1.8.4 GET – maltman Jun 30 '16 at 16:28
  • If there are places where a space may or may not be put a `( )?` which is an optional space – Will Barnwell Jun 30 '16 at 17:53
  • Thank you for the response Will. Doesn't seem to like that one either. I have also tried + but that did't work either. I don't know why the logs are saving like that. It is still just a single space. Maybe it is the format that they were sent – maltman Jun 30 '16 at 19:26
  • if its displaying like that it is definitely not just one space – Will Barnwell Jun 30 '16 at 19:35