3

Our team now uses zabbix for monitoring and alert. In addition, we use fluent to gather log to an central mongoDB and it is put to work for a week. Recently we were discussing another solution - Logstash. I wanna ask which difference between them? In my opinion, I'd like use zabbix as a data-gathering and alert-sending platform and fluent plays the 'data-gathering' role in the whole infrastructure. While I've looked into Logstash website and found out that Logstash is not only a log-gathering system, but also a whole solutions for gathering, presentation and search.

Would anybody can give some advice or share some experience?

Kazuki Ohta
  • 1,441
  • 17
  • 17
Frank Yao
  • 53
  • 1
  • 6

2 Answers2

2

Logstash is pretty versatile (disclaimer: have only been playing with it for a few weeks).

We'd been looking at Graylog2 for a while (listening for syslog and providing a nice search UI) but the message processing functionality in it is based on the Drools engine and is.. arcane at best.

I found it was much easier to have logstash read syslog files from our central server, massage the events and output to Graylog2. Gave us much more flexibility and should allow us to add application level events alongside the OS level syslog data.

It has a zabbix output, so you might find it's worth a look.

Rasputnik
  • 313
  • 2
  • 4
  • 1
    flume-ng's performance is far better than logstash so that I'm using flume now. – Frank Yao Jan 18 '13 at 11:02
  • If its working out for you, cool. Throughput in my case is less of a concern than the event filtering and transforms I can do with logstash, but as I say I'm dealing with a lot of different event inputs and formats. – Rasputnik Feb 24 '13 at 20:36
1

Logstash is a great fit with Zabbix.

I forked a repo on github to take the logstash statsd output and send it to Zabbix for trending/alerting. As was mentioned by another, logstash also has a Zabbix output plugin which is great for notifying/sending on matching events.

Personally, I prefer the native Logstash->Elasticsearch backend to Logstash->Graylog2(->Elasticsearch).

It's easier to manage, especially if you have a large volume of log data. At present, Graylog2 uses Elasticsearch as well, but uses only a single index for all data. If you're periodically cleaning up old data, this means the equivalent of a lot of SQL "delete from table where date>YYYY.MM.DD" type-calls to clean out old data, where Logstash defaults to daily indexes (the equivalent of "drop table YYYY.MM.DD"), so clean-up is nicer.

It also results in cleaner searches, requiring less heap space, as you can search over a known date because the index is named for the day's data it contains.

untergeek
  • 863
  • 4
  • 13
  • 1
    Hi Aaron? Small world it is. I'm using flume-ng to replace logstash. My data flow is 'file-flume-rabbitmq-flume' – Frank Yao Jan 18 '13 at 11:01
  • After testing of both, tend to agree with you. Add on Kibana as a UI and logstash is a simpler stack than graylog2, and so far is behaving better than our last graylog2 trial (0.9.6, 0.10.0 might have improved things). – Rasputnik Feb 24 '13 at 20:41