1

I have configured an elk stack (Logstash, Elastic search and and kibana)and we have a custom log file as below.

05/August/2015:16:55:10 : www.****.com : statuscode = 200 : Time in seconds to load = 0.734 05/August/2015:16:55:11 : ****.my : statuscode = 403 : Time in seconds to load = 0.340 05/August/2015:17:00:01 : www. ****.mx : statuscode = 200 : Time in seconds to load = 2.282 05/August/2015:17:00:03 : www. ****.my : statuscode = 200 : Time in seconds to load = 2.663 05/August/2015:17:00:06 : www. ****.co.id : statuscode = 200 : Time in seconds to load = 1.455 05/August/2015:17:00:08 : ****. ****.my : statuscode = 200 : Time in seconds to load = 1.684

I have configured this log succesfully on logstash and it is displaying properly in Kibana. Now I want to configure an email alert if any of the above website in the logs shows 504 or 403 status code for more than 5 count continously. I know I need to add a filter matching the pattern of the log file. But as this is a custom log, I’m unable to do it.

bytecode77
  • 14,163
  • 30
  • 110
  • 141
Manoj J
  • 21
  • 1
  • 3

4 Answers4

1

The best way will be to write you own script, I did it in Python.

The following is needed:

  • The script will be scheduled and search inside Elasticsearch for the result.
  • Once a result is found you will trigger a reaction.
  • If you have a monitoring system, you can send a trigger to this system to have one centralized interface.
  • If you just want an email you can use python to send an email with the result including the number of errors found, the server name and any other information that will help you debug the problem.
  • If you run the script every 10 minutes it should look in the Elasticsearch database for events in the past 10 minutes, if you run it once an hour then it should look for data from the past hour.
  • You can use cron for scheduling the script but you will need to find a way to monitor it in case of it failing, I use rundeck for this and it sends me an email if one of my jobs fails.

I know it might seem complex but if you are using your logstash infrastructure it will be worth writing this script and it will be fairly easy to adjust it to different searches once it is written.

Another option might be to wait for the watcher plugin by Elastic, from a post on their site they are planning to add a GUI interface to it once an official GA release will be out, I don't know when this will happen but it might be a better solution, even though it will be more limited than working with python.

Tom Kregenbild
  • 1,568
  • 1
  • 10
  • 11
1

Another project is elastalert.

It's open source. You can also find introductory blog posts: http://engineeringblog.yelp.com/2015/10/elastalert-alerting-at-scale-with-elasticsearch.html http://engineeringblog.yelp.com/2016/03/elastalert-part-two.html

Especially the Kibana rule convert should be very handy.

Christian
  • 1,487
  • 1
  • 14
  • 11
0

Since your condition is kind of complex (count of different response codes over time) i would dissuade from doing this with logstash.

However, this is exactly what the elasticsearch plugin Watcher is made for. You can use Watcher to search for specific response codes and specify conditions (e.g. count more than 5) and schedules to send an email.

I would recommend you to take a look at elastic's watcher introduction. I'm pretty sure it will suit your purpose.

hurb
  • 2,177
  • 3
  • 18
  • 32
0

You can use a service like Logz.io which helps you do exactly that. If you're shipping up to 1GB a day (30GB a month), it's free.

Tomer Levy
  • 357
  • 1
  • 4