Logstash Joining Field values in certain order after split.

Question

I am trying to parse appache accesslogs with logstash for a project.

To let elasticsearch search efficiënt I want to reverse the domainname. So for example:

www.example.com Becomes com.example.www

I tried to split the domainname and reverse that using the logstash mutate plugin. When parsed as is I get a field

domainname : "www.example.com"

Using settings below I get the following results:

Setting:

 mutate{
        split => {"domainname" => "."}
        add_field => {"reversed_domainname" => ["%{[domainname][-1]}","%{[domainname][-2]}"]}
        join => {"reversed_domainname" => "."}
}

Result:

domainname = [ "www", "example", "com" ]

reversed_domainname =[ "com", "example" ]

Everything works as intended untill the join function as seen in the results i dont get com.example

If I remove reversed from the join (split the domainname on "." and join them with "." I get the same results.

www.example.com becomes www.example.com

How can I join the fields in reversed order as it clearly should work.

baudsp · Answer 1 · 2016-12-07T11:07:33.627

0

For logstash Pre-5.x, try:

mutate{
    split => {"domainname" => "."}
}

ruby {
    code  => "event['domainname'] = event['domainname'].reverse"
}

The ruby filter allows you to execute ruby code, using a built-in function to reverse the array.

edited Dec 07 '16 at 11:07

answered Dec 06 '16 at 13:17

baudsp

4,076
1
17
35

however this should work i get the following error: Ruby exception occurred: undefined method `reverse' for nil:NilClass {:level=>:error} – JasperFennet Dec 07 '16 at 10:58
@JasperFennet Are you sure the `domainname` fields exists? Also, if you're using LS 5, use `event.get('domainname')` – baudsp Dec 07 '16 at 11:06
Im sure the field domainname exist. For testing pupose i tried the following code: x = event['domainname'].reverse print x This prints the array reversed as intended. but as soon as i assign it to a field i get the ruby exception (logstash 2.4.1) – JasperFennet Dec 07 '16 at 11:17
@JasperFennet It's odd since this message usually appears when you have an empty field/inexistant variable. I can't reproduce it and I don't have any idea how to solve it (except perhaps with a intermediate variable, like: `x = event['domainname'].reverse event['domainname'] = x`) – baudsp Dec 07 '16 at 12:41

score 0 · Accepted Answer · answered Dec 07 '16 at 14:28

Super derpy but using an intermediate variable fixed the problem. Don't know if this is ment or it's a bug. I used plugins below to fix my problem.

mutate {
        split => {"domainname" => "."}
   }
   ruby {
        code => "
                x = event['domainname']
                event['reversed_domainname'] = x.reverse.join('.')
                "
   }

   mutate{
        join => {"domainname" => "."}
   }

Logstash Joining Field values in certain order after split.

2 Answers2