logstash grok filter regular expression works in debug tool but failed in actual execution

Question

I'm trying to extract a filed out of log line, i use http://grokdebug.herokuapp.com/ to debug my regular expression with:

(?<action>(?<=action=).*(?=\&))

with input text like this:

 /event?id=123&action={"power":"on"}&package=1

i was able to get result like this:

{
  "action": [
   "{"power":"on"}"
 ]
}

but when i copy this config to my logstash config file:

input { stdin{} }

filter {
  grok {
    match => { "message" => "(?<action>(?<=action=).*(?=\&))"}
  }
}

output { stdout {
  codec => 'json'
}}

the output says matching failed:

{"message":" /event?id=123&action={\"power\":\"on\"}&package=1","@version":"1","@timestamp":"2016-01-05T10:30:04.714Z","host":"xxx","tags":["_grokparsefailure"]}

i'm using logstash-2.1.1 in cygwin. any idea why this happen?

Try using `\baction=(?[^&]*)` or even `[?&]action=(?[^&]*)`. The negated character class `[^&]` will make it less greedy, and should be more efficient and potentially solve other issues. — Wiktor Stribiżew, Jan 05 '16 at 10:35
thank you, this works works for me, as you said by making it less greedy, will help avoid lots of problems — Gavin Huang, Jan 06 '16 at 01:29
Please consider also upvoting my answer if it proved helpful to you. — Wiktor Stribiżew, Jun 15 '17 at 08:39

score 0 · Answer 1 · answered Jan 05 '16 at 14:51

0

It doesn't answer your regexp question, but...

Parse the query string to a separate field and use the kv{} filter on it.

answered Jan 05 '16 at 14:51

Alain Collins

16,268
2
32
55

Thank you for offer this approach, i tried to match the query string by %{URIPATH} but turns out it can't capture the string starts with "{" – Gavin Huang Jan 06 '16 at 01:28
The curly brace is allowed in URIPATH; it's the double-quote that's not part of that pattern. Use a different pattern. – Alain Collins Jan 06 '16 at 03:35

score 0 · Accepted Answer · answered Jan 06 '16 at 08:47

0

You might experience an issue caused by a greedy dot matching subpattern .*. Since you are only interested in a string of text after action= till next & or end of string you'd better use a negated character class [^&].

So, use

[?&]action=(?<action>[^&]*)

The [?&] matches either a ? or & and works as a boundary here.

answered Jan 06 '16 at 08:47

Wiktor Stribiżew

607,720
39
448
563

Please consider also upvoting my answer if it proved helpful to you. – Wiktor Stribiżew Jan 09 '16 at 08:10

logstash grok filter regular expression works in debug tool but failed in actual execution

2 Answers2