How to remove underscores in field names with logstash?

Question

I'm thinking about using the mutate filter and the rename option, but I don't know about the corresponding regex to achieve that:

filter {
  mutate {
    rename => {
      "any_field_with_underscore" => "anyfieldwithunderscore" # i don't know how to write regex for this ...
    }
  }
}

Can anyone help?

Alain Collins · Answer 1 · 2016-01-12T17:31:57.477

6

There no indication in the doc that rename{} takes a regexp.

I've seen this done with a ruby{} filter.

As requested, here's some untested Ruby:

begin
    keys = event.to_hash.keys
    keys.each{|key|
        if ( key =~ /_/ )

            newkey = key.gsub(/_/, '')
            event[newkey] = event.remove(key)

        end
    }

rescue Exception => e
    event['logstash_ruby_exception'] = 'underscores: ' + e.message
end

edited Jan 12 '16 at 17:31

answered Jun 29 '15 at 15:54

Alain Collins

16,268
2
32
55

okay... well just to be a bit greedy how would one accomplish it with ruby{}? – lingxiao Jun 30 '15 at 05:43
It almost works. Not a big deal to store the kvs and add them outside of the iteration: Ruby exception occurred: can't add a new key into hash during iteration {:level=>:error, :file=>"logstash/filters/ruby.rb", :line=>"41", :method=>"filter"} – Todd Flanders Feb 06 '16 at 13:37
That's why it copies the keys to a new variable before the loop. Maybe you took that out of your code? – Alain Collins Feb 06 '16 at 18:10
Works perfectly as is for LS < 5.x, But note that direct access to the event object will not work in newer versions because it has been replaced with a pure Java object, and will require access via getter and setter. – PhaedrusTheGreek Jun 03 '16 at 13:44

score 1 · Answer 2 · edited May 23 '17 at 12:06

To build on Alain's answer,

In Logstash >= 5.x, an event object accessor is enforced:

 ruby {

  code => "
        begin
            keys = event.to_hash.keys
            keys.each{|key|
                if ( key =~ /http_/ )

                    newkey = key.gsub(/http_/, '')
                    event.set(newkey, event.remove(key))

                end
            }

        rescue Exception => e
            event.set('logstash_ruby_exception', 'underscores: ' + e.message)
        end
        "
 }

}

Also see this feature request that would do the same

How to remove underscores in field names with logstash?

2 Answers2

Linked