-1

I'm getting a consistent grok failure on brackets. I've escaped the bracket with a '\'. I've also tried to fool it with wildcards. Grok debugger parses correctly, but fails to parse in production. I have two other entries that are exhibiting the same behavior while other patterns without brackets are parsing.

The '%{WORD:type}[%{INT:pid}]' is the issue.

#Jan 15 13:35:44 firewall sshd[1468]: Accepted publickey for john from 192.168.1.16 port 62529 ssh2: .....
AUTHLOG1 (%{SYSLOGTIMESTAMP:timestamp} %{WORD:src_host} %{WORD:type}\[%{INT:pid}\]: Accepted publickey for %{USERNAME:user} from %{IP:src_ip} port %{INT:port} %{WORD:protocol}*)

#Jan 15 13:35:44 firewall systemd-logind[1221]: New session 481 of user john.
AUTHLOG4 (%{SYSLOGTIMESTAMP:timestamp} %{WORD:src_host} (?<type>[a-z-]+)\[%{INT:pid}\]: Removed session %{INT:session}.)

AUTHLOG (?:%{AUTHLOG1}|%{AUTHLOG2}|%{AUTHLOG3}|%{AUTHLOG4}|%{AUTHLOG5}|%{AUTHLOG6})

 if [type] == "authlog" {
        grok {
                match => { "message" => "%{AUTHLOG}"}
                patterns_dir => ["/etc/logstash/grok"]
        }
  }
Michael Johnson
  • 11
  • 2
  • Please take the [Tour](https://stackoverflow.com/tour), read [How To Ask](https://stackoverflow.com/help/how-to-ask) and create an [MCVE](https://stackoverflow.com/help/mcve). – Hermann Döppes Jan 15 '17 at 19:51
  • I'm getting a ' tags:_grokparsefailure' in Kibana. Logstash isn't parsing it correctly. – Michael Johnson Jan 16 '17 at 02:47

2 Answers2

0

You're biting off a lot with that config.

First, try pulling out the syslog stuff first. It's common to each line, so it's easy to match and pull off of the line, leaving only the more unique stuff to be parsed by the other patterns. In other words, don't do "date A|date B|date C" but pull of the date and then look for A, B, C. Shorter strings run against shorted regular expressions should be better.

Next, imagine what the underlying regexp looks like that you're trying to use. I stay away of "|" except in very small patterns ("On|Off" type of things). I haven't measured the performance, but I prefer to see it written:

match => { message => [
    "Pattern 1 %{WORD} ...",
    "Pattern 2 %{WORD} ..."
    ]
}

To start answering your real question, brackets in regular expressions are used to show character classes, e.g. "[a-z]". To tell grok that you mean a literal bracket, you need to escape only the open bracket, e.g. "\foo]" (as the closing bracket has lost its magic by not being paired with an opening brakcet). It should be fine to escape both for readability, e.g "\[foo\]".

Your example contains 6 patterns that you're applying to each input line. The grokparsefailure only tells you that none of them matched. To find out why, look at the single pattern you expected to match. If the error is not obvious, take the pattern and your input line to the grok debugger. Chop off the 2nd half of the pattern and see if it matches. Repeat until you find the part that breaks.

In your case, just pasting the pattern into the debugger made me wonder why you had parenthesis around the pattern when none appear in the input. Take those out (and the asterisk at the end), and I think it matches.

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
0

OK, I figured it out. It had nothing to do with how I escaped brackets. I used the reserved word 'type' as a variable.

Michael Johnson
  • 11
  • 2