Regex in config for dynamic columns in logstash

Question

I have the log file of which i have pasted two rows below:

Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=traffic subtype=forward level=notice vd=root srcip=172.16.1.251 srcport=62032 srcintf="Combo_LAN" dstip=X.X.X.X dstport=X dstintf="wan2" sessionid=16172588 status=close user="X.X" group="Open Group" policyid=2 dstcountry="United States" srccountry="Reserved" trandisp=snat transip=X.X.X.X transport=X service=HTTP proto=6 applist="Block_Applications" duration=11 sentbyte=2377 rcvdbyte=784 sentpkt=6 rcvdpkt=7 identidx=5 utmaction=passthrough utmevent=webfilter utmsubtype=ftgd-cat urlcnt=1 hostname="tacoda.at.atwola.com" catdesc="Advertising"

Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=utm subtype=webfilter eventtype=ftgd_allow level=notice vd="root" policyid=2 identidx=5 sessionid=15536743 user="X.X" srcip=X.X.X.X srcport=X srcintf="Combo_LAN" dstip=X.X.X.X dstport=80 dstintf="wan2" service="http" hostname="streaming.sbismart.com" profiletype="Webfilter_Profile" profile="Open Group_Policy" status="passthrough" reqtype="direct" url="/diffusion/" sentbyte=984 rcvdbyte=202 msg="URL belongs to an allowed category in policy" method=domain class=0 cat=18 catdesc="Brokerage and Trading"

My question is i can parse the data if number of columns and order is fixed.

But, how do i parse the dynamic columns in the config file so that i don't get the _grokparsefailure?

It's very hard to read these rows here. Also, two rows are hardly sufficient to gather how dynamic these columns can be. Can you describe what exactly is causing the error? — Tim Pietzcker, Nov 27 '14 at 08:04
What i am saying is order of columns is also not fixed. As you can eventtype column was not there in row 1 but it was present in the row 2. So, how do i handle it? — Naresh, Nov 27 '14 at 08:45
Exactly what do you mean by "parse"? Do you mean "marshal" a dynamic object using the key=name pairs in a flexible manner? — Bohemian, Nov 28 '14 at 06:25
Yes, Bohemain i want to marshal a dynamic object using key=value pairs in a flexible manner. — Naresh, Dec 01 '14 at 07:25

Ban-Chuan Lim · Accepted Answer · 2014-12-01T09:47:16.860

1

Ruby Plugin can help you.

Here is the configuration:

input {
    stdin{
    }
}

filter {
    ruby {
        code => '
            msg = event["message"]
            msgIndex = msg.index("date=")
            msgInsert = msg[msgIndex..-1]
            msgMap = msgInsert.scan(/(\w+)=("(.*?)"|([^ ]+))/).map { |(first, second)| [first, second] }
            for x in msgMap
                key = x[0]
                value = x[1]
                event[key] = value
            end
        '
    }
}

output {
    stdout{
        codec => rubydebug
    }
}

First, get all the key=value pair by index the start value date=
Then map all the key,value to string array.
Use For loop to insert all the value.

I have try your logs and I can create all the correspond field with the value. Hope this can help you

edited Dec 01 '14 at 09:47

answered Nov 28 '14 at 04:52

Ban-Chuan Lim

7,840
4
35
52

I think if you want to take the first word only maybe you have to filter it in the for loop. For example, split the string by "space" and take the first word. – Ban-Chuan Lim Dec 01 '14 at 00:21
No, i am not saying i want to take only first word. Currently regex is working in such a way that it only giving you the first word not the whole sentence. So, could you please tell me how to take to whole string not the first word. – Naresh Dec 01 '14 at 07:22
OK! I have updated the answer! Add the ? and change the order of the regexp. I have tested it again with your logs, it can parse out all the sentence. – Ban-Chuan Lim Dec 01 '14 at 09:48

score 1 · Answer 2 · answered Nov 29 '14 at 08:26

The simple answer to avoiding grokparsefailure is to provide a valid pattern that matches your input. That said, your question seems to imply that the fields are not always specified in this order. Given the examples, you should be using the "kv" filter to split these key/value pairs into fields.

Regex in config for dynamic columns in logstash

2 Answers2