Can I delete the message field from Logstash?

Question

I have a basic Logstash -> Elasticsearch setup, and it turns out the 'message' field is not required after the logstash filter done its job - storing this raw message field to elasticsearch is only adding unnecessary data to storage imo.

Can I safely delete this field and would it cause any trouble to ES? advices or readings are welcome, thanks all.

Ban-Chuan Lim · Accepted Answer · 2015-07-27T01:06:00.460

37

No, it will not cause any trouble to ES. You can delete message field if it is redundant or unused.

You can add this filter to end of the filters.

mutate
{
     remove_field => [ "message" ]
}

edited Jul 27 '15 at 01:06

answered Sep 24 '14 at 01:16

Ban-Chuan Lim

7,840
4
35
52

3

ProTip: Most filters (such as grok) also allow the remove_field option, and will only remove the field if the filter does not encounter an error. That can be both safer and easier than using an additional mutate filter. – Brian Papantonio May 11 '17 at 05:58

score 11 · Answer 2 · answered Sep 10 '16 at 20:53

11

You can also do this within the json filter.

filter {
  json {
    source => "message"
    remove_field => ["message"]
  }
}

answered Sep 10 '16 at 20:53

Steve

6,618
3
44
42

this one is crashing my kafka output , only the remove_field part, not sure why :( – llermaly Dec 24 '20 at 08:58

Graham Hannington · Answer 3 · 2017-03-07T08:09:04.947

I would have added the following as a comment to the answer by Ben Lim, but I do not know how to add a code block in a comment, or even whether that is possible...

If you can use a combination of input and codec that does not create a message field, then you do not need to remove it.

For example, the following combination of input and codec (JSON Lines over TCP) does not create a message field:

input {
  tcp {
    port => 5044
    codec => json_lines
  }
}
output {
  elasticsearch {
    hosts => ["localhost"]
    document_type => "mytype"
    index => "myindex"
  }
}

Can I delete the message field from Logstash?

3 Answers3