0

I'm building a ELK Setup and its working fine , however i'm getting into a situation where i want to remove certain fields from by system-log data while processing through logstash like remove_field & remove_tag which i've defined in my logstash configuration file but that's not working.

Looking for any esteem and expert advice to correct the config to make it running, thanks very much in advanced.

My logstash configuration file:

[root@sandbox-prd~]# cat /etc/logstash/conf.d/syslog.conf
input {
  file {
    path => [ "/data/SYSTEMS/*/messages.log" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    max_open_files => 64000
    type => "sj-syslog"
 }
}

filter {
  if [type] == "sj-syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp } %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => ["@version", "host", "_type", "_index", "_score", "path"]
      remove_tag => ["_grokparsefailure"]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
  }
 }
}
output {
        if [type] == "sj-syslog" {
        elasticsearch {
                hosts => "sandbox-prd02:9200"
                manage_template => false
                index => "sj-syslog-%{+YYYY.MM.dd}"
                document_type => "messages"
  }
 }
}

Data sample appearing on the Kibana Portal

syslog_pid:6662 type:sj-syslog syslog_message:(root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) syslog_severity:notice syslog_hostname:dbaprod01 syslog_severity_code:5 syslog_timestamp:Feb 11 10:25:02 @timestamp:February 11th 2019, 23:55:02.000 message:Feb 11 10:25:02 dbaprod01 CROND[6662]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) syslog_facility:user-level syslog_facility_code:1 syslog_program:CROND received_at:February 11th 2019, 10:25:03.353 _id:KpHo2mgBybCgY5IwmRPn _type:messages
_index:sj-syslog-2019.02.11 _score: -

MY Resource Details:

OS version : Linux 7

Logstash Version: 6.5.4

Karn Kumar
  • 8,518
  • 3
  • 27
  • 53

1 Answers1

2

You can't remove _type and _index, those are metadata fields needed by elasticsearch to work, they have information about your index name and the mapping of your data, the _score field is also a metadata field, generated at search time, it's not on your document.

leandrojmp
  • 7,082
  • 2
  • 19
  • 24
  • @- leandrojmp, many thanks for your expertise answer, let me check the consideration..However, whats about `remove_tag => ["_grokparsefailure"]` – Karn Kumar Feb 11 '19 at 13:39
  • It will remove a tag named `_grokparsefailure` if your grok filter is successful. But since you only have one grok filter, there is no need to have this line, it would make sense if you had another grok and wanted to remove the tag after a failed grok. – leandrojmp Feb 11 '19 at 13:45
  • Thnx a lot it works :-) , new thing learned, appreciate your all help. – Karn Kumar Feb 11 '19 at 13:57