Filter specific Message with logstash before sending to ElasticSearch

Question

I had like to know if it is possible to send only specific log messages to elasticsearch via logstash? E.G let's say I have these messages in my log file:

2015-08-14 12:21:03 [31946] PASS  10.249.10.70  http://google.com
2015-08-14 12:25:00 [2492]  domainlist \"/etc/ufdbguard/blacklists\
2015-08-14 12:21:03 [31946] PASS 10.249.10.41 http://yahoo.com

I had like to skip the second line when logstash/log forwarder process this log, is it possible to instruct it to skip any log message with the keyword 'domainlist'? Or allow only log messages with the keyword 'PASS'?

score 2 · Accepted Answer · answered Sep 03 '15 at 08:15

Yes, you can achieve that by using the drop filter.

Depending on how you grok your log line and which field names you have, you can decide to drop an event if it matches some criteria. For instance, below you can see a conditional after the grok filter, which checks whether myfield contains something different than the value PASS in which case it will drop the event.

filter {
  grok {
      ...your parsing regexp here...
  }

  if [myfield] != "PASS" {
    drop { }
  }
}

score 1 · Answer 2 · answered Sep 10 '15 at 05:48

1

Although Val's answer is correct, another way you can do it is by filtering with a conditional:

output {
  if "PASS" in [message] {
    elasticsearch {
      ...
    }
  }
}

answered Sep 10 '15 at 05:48

Hyder B.

10,900
5
51
60

Filter specific Message with logstash before sending to ElasticSearch

2 Answers2

Linked