logstash grok - how to do conditional pattern matching?

Question

I have these three log lines in the same log file:

INFO [2015-08-27 18:46:14,279] ({qtp243745864-44} NotebookServer.java[onMessage]:101) - RECEIVE << RUN_PARAGRAPH
INFO [2015-08-27 18:46:14,322] ({qtp243745864-44} NotebookServer.java[broadcast]:253) - SEND >> NOTE
INFO [2015-08-27 18:46:16,809] ({pool-1-thread-2} RemoteInterpreter.java[init]:144) - Create remote interpreter org.apache.zeppelin.markdown.Markdown

I want to pars them using grok but failing to get the right fields: 1) how to pars the data within the brackets? 2) the last part of the log line is either (CMD direction cmd_data) or (cmd info) in the example:

cmd=Receive or SEND
cmd_direction=<< or >>
cmd_data=RUN_PARAGRAPH or NOTE

But the last line is CMD info which does not correspond to the same format.

I am trying to find the right rule that will match the first and second but not the third. end result should be or (cmd + cmd_data) or (cmd_info) fields Any help?

score 12 · Answer 1 · answered Sep 03 '15 at 13:18

12

Logstash has conditionals in the config file, so you can conditionally match things.

For example:

if ([mesage] =~ /(RECEIVE|SEND)/) {
   grok {
      // do your grok here
   }
} else if ([message] =~ /RemoteInterpreter/) {
   grok {
      // do some other grok here
   }
}

If you need help with what those groks should be, try using the grok debugger

answered Sep 03 '15 at 13:18

Alcanzar

16,985
6
42
59

Can this be done on a regex? e.g. if ([mesage] =~ /(^\S+ << ^\S+ | ^\S+ >> ^\S+) – Eran Witkon Sep 03 '15 at 15:59
you can use a regex with the if... but the `^` placement is wrong -- its the anchor for start of message. So you'd want something like `(\S+ (<<|>>) \S+)` – Alcanzar Sep 03 '15 at 16:15
Thanks! Will test it soon – Eran Witkon Sep 03 '15 at 16:20

score 0 · Answer 2 · answered Jul 31 '17 at 09:21

0

I had a similar problem ...

It helped me:

if ("SEND" in [message]) {
   grok {
      // do your grok here
   }
}

answered Jul 31 '17 at 09:21

vvator

168
2
11

logstash grok - how to do conditional pattern matching?

2 Answers2