0

I currently have two machines that receive about 1000 HTTP req/sec both and generate a log entry on every request. This log is centralized on a syslog daemon running on a different machine.

For reasons not really relevant, I will need to have those logs sent to an internal pub/sub server (each log entry will be a JSON payload sent over UDP), for further treatment in [near] real-time.

I was thinking about keeping everything up until the syslog daemon, because I still need those logs centralized, but was wondering how I could take it from there to the pub/sub server. A few things currently crossing my mind are not particularly brilliant:

  • use syslog-ng with a pipe as a destination and have a small process picking up stuff from there, processing it and sending it via UPD
  • have a small process tailing the centralized log file, processing new entries and sending them via UPD.
  • …

I'd love to hear some more experienced folks that might have had a similar problem. Ideally syslogd would receive logs from the different machines, perform its required transformations to JSON, send it over UPD and then also write it locally to the centralized log file. Everything is really debatable and opened to discussion, since there might be other solutions I never even heard of.

There's not really a requirement for performance here (1000 req/sec is a really small scale). The only real requirement here is that it should be as close to real-time as possible (about 10 seconds delay or something is acceptable).

Pedro Mata-Mouros
  • 3
  • 1

1 Answers1

0

syslog-ng is (IMHO) the correct way to do this. Once you receive the message, send it to a Perl script using the program() destination in syslog-ng. If you want to search and chart/graph after parsing it, try a tool like LogZilla that will work in-line with what you are trying to do. Perl can, without any special work, process up to around 35k events per second, so you are good there. Check here for an overview picture of how this is done in LZ (which also uses a Perl program destination).

Clayton Dukes
  • 444
  • 2
  • 9