0

I have created a demo environment using Logstash, redis , elastic search and kibana. (http://indico.cern.ch/getFile.....

Here logstash shipper is reading logs from log file which i have centralized using syslog-ng. Loogstash shipper is forwarding it to redis then Logstash indexer (filter) and finally to Elasticsearch.

Now i want to skip logstash shipper and redis part from it. Is this a good idea? Or redis is mandatory Or require to deal with heavy load. I'm not sure about it.

In above pdf link i have read that Logstash has low buffering and redis manages flow of logs that why redis is used. As redis keeps data in memory what if memory gets full? Also read that Logstash and Elasticsearch can be quite hungry in terms of RAM usage. JVM options needs to be properly tuned. if so then, how to tune JVM?

Is it required to purge/rotate elasticsearch data/index?

So which one is best suited for heavy load? I want to parse logs like [ System (OS and daemons ) logs, syslog, web server logs (apache, lighttpd), application server logs (tomcat), database server logs (mysql) and some Application logs (through logfiles) ].

Give your suggestions for improvement. Thanks !!!.

Kindly find following link for IMAGE.

(http://a.disquscdn.com/uploads/mediaembed/images/709/3604/original.jpg)

amolnavghane23
  • 1
  • 3

1 Answers1

0

In the set up you describe Redis should not be required, using syslog-ng to centralise the log files serves the same purpose as Redis when multiple shippers are used.
It might be necessary to prune elasticsearch indexes to reduce disk space requirements. This will depend on how quickly your elasticsearch data are growing, how much disk space you have available and how long you need the logs to be searchable for.

I can't advise on JVM tuning.

user1013341
  • 336
  • 1
  • 9