0

I have some troubles filtering my logs using logstash because my pattern only works in the debugger

FilterString

127.0.0.1 - - [06/Jan/2016:15:43:41 +0000] "GET /index.php/banner/ajax/load/?sections=&_=1452095017076 HTTP/1.1" 200 74 "https://magento2-dev.argento.io/index.php/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "-"

My Pattern

%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}

When I use this in http://grokdebug.herokuapp.com/ everything is just fine, but it breaks when I use it in my config.

in logstash conf

filter {
  grok {
    match => [ 'message', '%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}' ]
  }
}

I strongly belive this is related to quotes or slashes. I tried with single quotes, but no luck here.

thanks for helping me out

xhallix
  • 2,919
  • 5
  • 37
  • 55
  • "it breaks" means what? logstash doesn't start? you get grokparsefailures? or ? – Alain Collins Jan 06 '16 at 18:53
  • sorry that means no index pattern – xhallix Jan 06 '16 at 20:21
  • If no index is being created, check the logstash and elasticsearch logs. If you don't find a good clue there, add a stdout{} output stanza to your logstash config and make sure it looks like what you're expecting. – Alain Collins Jan 06 '16 at 21:50
  • @AlainCollins thanks for the reply. The issue is coming from the grok filter I'm using and I wonder why it is not working while it does in the debugger – xhallix Jan 06 '16 at 21:54
  • I can't say that I understand what "no index pattern" means either. Can you describe the problem in more detail (an error message, a field mismatch, etc?) and where it exists (logstash, elasticsearch, kibana)? – Alain Collins Jan 07 '16 at 05:16
  • I visualize my errors which were sent to logstash in kibana. therefore I use some filters logstash. If the filter is not correct in logstash then I have no default pattern to match against in kibana. – xhallix Jan 07 '16 at 06:19
  • try to specify the patterns directory explicitly in your filter. One possible reason could be that some patterns are not included by your logstash version but are being included in the online debugger. That is why it does not work on your logstash version. This is just speculation, I can help you more if you post the STDOUT, error messages of ES or Logstash, etc. – Mrunal Pagnis Jul 12 '16 at 07:50

0 Answers0