Logstash grok chain conditional filters

Question

I'm trying to create a grok pattern for a mixed log. This is my first time creating a conditional chain and I keep getting syntax errors:

opt/logstash/bin/logstash -f /opt/logstash/conf.d/sip-parser.conf --    configtest
Error: Expected one of #, in, not , ==, !=, <=, >=, <, >, =~, !~, and, or,    xor, nand, { at line 27, column 14 (byte 580) after filter {
    # separate soap calls from responses
            grok {
                    match => { "message" => "\[%{TIMESTAMP_ISO8601:logdate}  \] %{LOGLEVEL:level}   %{GREEDYDATA:type}"}
                    }
                    if [type]

My configfile:

input {
    file{
            path => "/home/steven/sip.log"
            start_position => beginning
            # logstash stores the lastrun=> so we trick it
            sincedb_path => "/dev/null"

            #if logentry does not start with date it's part of previous entry
            codec =>  multiline {
            pattern => "\[^%{TIMESTAMP_ISO8601:logdate}\]"
            negate => "true"
            what => "previous"
    }

            }
    }

filter {
            grok {

                    match => { "message" => "\[%{TIMESTAMP_ISO8601:logdate}  \] %{LOGLEVEL:level}   %{GREEDYDATA:type}"}
                    }
                    # separate soap calls from responses
                    if ([type] ~= /AbstractLoggingInterceptor:\ Inbound Message$/) {
                            grok {
                                    match => { "message" => "\[%{TIMESTAMP_ISO8601:logdate}  \] %{LOGLEVEL:level}   %{GREEDYDATA:type}\n----------------------------\n%{GREEDYDATA:id}\n%{GREEDYDATA:responsecode}\n%{GREEDYDATA:encoding}\n%{GREEDYDATA:contenttype}\n%{GREEDYDATA:headers}\n%{GREEDYDATA:payload}\n--------------------------------------"}
                            }
                    }
                    else if ([type] ~= /AbstractLoggingInterceptor:\ Outbound Message$/) {
                            grok {
                                    match => {"message" => "\[%{TIMESTAMP_ISO8601:logdate}  \] %{LOGLEVEL:level}   %{GREEDYDATA:type}\n---------------------------\n%{GREEDYDATA:id}\n%{GREEDYDATA:responsecode}\n%{GREEDYDATA:encoding}\n%{GREEDYDATA:contenttype}\n%{GREEDYDATA:headers}\n%{GREEDYDATA:payload}\n--------------------------------------"}
                            }
                    }
                    else {
                            grok {
                                    match => {"message" => "\[%{TIMESTAMP_ISO8601:logdate}  \] %{LOGLEVEL:level}   %{GREEDYDATA:type}"}
                            }
                    }

            }

output {
    #elasticsearch {}
    stdout{}
}

The logfile I'm trying to parse can be found here: http://pastebin.com/afbNfmjW The individual grok patterns for each different type of entry have been tested in http://grokdebug.herokuapp.com/ but I can't chain these together. What am I doing wrong?

score 2 · Accepted Answer · answered Jan 11 '16 at 14:51

2

Your conditional grok{}s should not be inside the first grok, but peers to it:

grok { ... }
if [myField] == "value" {
    grok { ... }
}

Also note that you're running a regular expression to see if you should run a regular expression. I would suggest sending multiple patterns to one grok stanza:

grok {
    match => { "myField",
        pattern1,
        pattern2,
        pattern3
    }
}

by default, grok will stop processing them when one matches.

answered Jan 11 '16 at 14:51

Alain Collins

16,268
2
32
55

Thanks, this works. The only issue I still have is that logstash only reads the first entry of the log and then stops parsing. – user211984 Jan 19 '16 at 12:11

Logstash grok chain conditional filters

1 Answers1