0

Im at a loss and probably should step away from the problem, Can anyone help spot what I am missing. Logstash keep thowing "_grokparsefailure". Scratching my head???

using logstash logstash-1.3.3-flatjar.jar

Log file example

proxy.ian.com - ian@IAN.COM [24/Feb/2014:11:16:49 -0500] "GET /docs/en-US/Guide/+ HTTP/1.1" 404 285 "https://ian.com/docs/en-US/Guides/html/Guide" "Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0"

My logstash filter

Filter {
   if [type] == "ErcAccess" {
    grok {
    match => ["message", "%{IPORHOST:clientip} - %{USER:auth}@%{URIPROTO}.%{WORD:domain} \[%{HTTPDATE:timestamp}\] "%{WORD:httpmethod} %{NOTSPACE:referrer} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "%{NOTSPACE:request}" %{QS:UserAgent}" ]
         }
     }
}
IanN
  • 289
  • 4
  • 13

1 Answers1

4

Your pattern includes " characters, which are apparently treated as literal characters by grokdebug. When Logstash is reading your config file, those quote characters have a different semantic meaning (they mark the beginning or ending of a string).

UPDATE: turns out Logstash's escaping of quotes is poorly documented and possibly buggy. I'll update if I find a better solution, but for now it looks like you can use ' single quotes to begin/end your strings, which will allow you to use " double quotes freely within them.

This works for me:

input {
    generator {
        type => 'ErcAccess'
        message => 'proxy.ian.com - ian@IAN.COM [24/Feb/2014:11:16:49 -0500] "GET /docs/en-US/Guide/+ HTTP/1.1" 404 285 "https://ian.com/docs/en-US/Guides/html/Guide" "Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0"'
        count => 1
    }
}

filter {
  if [type] == 'ErcAccess' {
    grok {
      match => ['message', '%{IPORHOST:clientip} - %{USER:auth}@%{URIPROTO}.%{WORD:domain} \[%{HTTPDATE:timestamp}\] "%{WORD:httpmethod} %{NOTSPACE:referrer} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "%{NOTSPACE:request}" %{QS:UserAgent}' ]
    }
  }
}

output {
    stdout {
        codec => rubydebug{}
    }
}
rutter
  • 11,242
  • 1
  • 30
  • 46
  • Yep I thought that too and escaped the chars, but still get grokparsefailures. I even removed the " from the log files as an attempt..this also returned the same parse failure issue. – IanN Mar 18 '14 at 06:45
  • @IanN I've updated my answer with a better solution. I'm still trying to find the best general-case answer, but the above works for the sample message you gave. – rutter Mar 24 '14 at 23:04