Filebeat/Logstash remove unwanted fields & values from output

Question

My Filebeat configuration is very simple -

- input_type: log
  paths:
    - C:\log\FilebeatInputTest.txt

output.logstash:
  hosts: ["http://X.X.X.X:XXXX"]

if I write something in ilebeatInputTest.txt like - This is from Filebeat

I get output in Elastic search something like - ....... "index": "logstash-" "source" : { "@timestamp": "2017-05-19T06:41:02.663Z", "beat": { "hostname": "CHITTARS02", "name": "CHITTARS02", "version": "5.4.0" }, "input_type": "log", "message": "This is from Filebeat", "offset": 23, "source": "C:\\log\\FilebeatInputTest.txt", "type": "log" } .....

My pipeline is Filebeat(monitoring FilebeatInputTest.txt) > Logstash > Elasticsearch

logstash.cnf as follows -

input {

    beats {
        port => 25000
    }
}
output {

    elasticsearch {
        hosts => ["http://xx.xx.xx.xx:XX"]
        user => "elastic"
        password => "changeme"
    }
}

Problem : Can I remove all unwanted keys & values from output? That is, I want my output should be something like -

....... "index": "logstash-" "source" : { "message": "This is from Filebeat", } ......

I want to remove "@timestamp", "beat","input_type""offset","source","type"

I tried with following -

filter{
    prune {
        blacklist_names => ["@timestamp", "beat","input_type""offset","source","type"]
    }

}

And

filter{
    mutate {
        remove_field => ["@timestamp", "beat","input_type""offset","source","type"]
    }
}

But no help, results are same

I have been looking for something at the filebeat end before send. have you had any luck? — AnthonyJClink, Aug 25 '21 at 06:50

score 2 · Answer 1 · answered Jun 01 '17 at 15:10

2

You're using the correct method, but there's a typo in your remove_field list. You missed a comma. It should be:

filter{
    mutate {
        remove_field => [ "@timestamp", "beat", "input_type", "offset", "source", "type" ]
    }
}

answered Jun 01 '17 at 15:10

Jason Crease

1,896
17
17

score 2 · Answer 2 · answered Jan 17 '20 at 11:35

2

Another solution is to remove these fields with filebeat.

processors:
  - add_host_metadata: ~
  - drop_fields:
    fields: ["type", "@version", "offset", "tags"]

answered Jan 17 '20 at 11:35

Ivan Vovk

929
14
28

score 0 · Answer 3 · answered May 19 '17 at 12:29

0

May guess is that you forget to put the port in quotes; that is instead of 25000 used "25000". Try this

input {

    beats {
        port => "25000"
    }
}

filter{
    mutate {
        remove_field => ["@timestamp", "beat","input_type","offset","source","type","@version","host","tags"]
    }
}

output {

    elasticsearch {
        hosts => ["http://xx.xx.xx.xx:XX"]
        user => "elastic"
        password => "changeme"
    }
}

Input

This is from Filebeat

Output

{
    "_index" : "logstash-",
    "_type" : "logs",
    "_id" : "AVwglLbLfqaeaIoZluvE",
    "_score" : 1.0,
    "_source" : {
      "message" : "This is from Filebeat"
    }
}

I also removed the fields "@version","host" and "tags".

Hope this helps.

answered May 19 '17 at 12:29

berrytchaks

839
10
18

Hi, thanks for quick response, but, if i use - filter{ mutate { remove_field => ["offset", "input_type", "beat", "host", "source", "type", "tags"] } } i get this o/p - { "_index" : "logstash-2017.05.31", "_type" : "logs", "_id" : "AVxem68fw204zcLHaGHa", "_score" : 1.0, "_source" : { "@timestamp" : "2017-05-31T13:04:37.730Z", "@version" : "1", "message" : "Aare Samba kitna admi tha?" } } – CR Sardar May 31 '17 at 13:19
But when i add "@timestamp", "@version", as - filter{ mutate { remove_field => ["offset", "input_type", "beat", "host", "source", "type", "tags", "@timestamp", "@version"] } } Logstash is exiting with following error - An unexpected error occurred! {:error=>#, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:202:in `sprintf'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/common.rb:153:in `event_action .................. – CR Sardar May 31 '17 at 13:20
In all cases my input is as follows - { "_index" : "logstash-2", "_type" : "log", "_id" : "AVxeaCoN", "_score" : 1.0, "_source" : { "@timestamp" : "2017-05-31T10:23:53.436Z", "offset" : 11, "@version" : "1", "input_type" : "log", "beat" : { "hostname" : "elk-", "name" : "elk-chitta", "version" : "5.4.0" }, "message" : "Hi Elastic", ............. } } – CR Sardar May 31 '17 at 13:26

Filebeat/Logstash remove unwanted fields & values from output

3 Answers3