-1

I need a "thing" sitting on a host listening to text logs from some source (fluentd) which I can query for:

  • stored logs for a given time window
  • stored logs with simple filtering (akin to grep)
  • live feed (akin to tail -f)

My current idea is to have fluentd to send logs to a plain old RDBMS (Postgres) to achieve the first two, but I don't know of any ready-made solution for the third one. I'm not beyond writing one myself, but I can't believe nobody did this already :-)

Also, feel free to suggest a different storage, I'm not married to Postgres, it's just something off the top of my head.

isagalaev
  • 1,173
  • 11
  • 16

1 Answers1

1

You may want to try the MongoDB output plugin writing into capped collections, which support a kind of tail -f with tailable MongoDB cursors.

They even have a script mongo-tail to help with that in the GitHub plugin repo.

When using this with the latest MongoDB plugin over Fluentd v0.12 – which has some handy features over the stable Fluentd – you may have to apply a bug fix (which is waiting to be applied upstream) to make sure capped collections are correctly created and tailing works fine – see https://github.com/fluent/fluent-plugin-mongo/pull/91.

A. Ferreira
  • 31
  • 3