Fluentd persistence and reliability

Question

Since fluentd does not use redis but supposedly has better built in reliability, how does that solve the problem of the instance going down before it has a chance to send the logs to elastic search? Is this something not significant enough to worry about, for example you could set the steaming of logs at a high frequency, so if you ever lose the instance, only a few lines would have no transferred over?

Answer 1

Fluentd uses a buffering mechanism, once it receive a set of events they are stored either in memory or in the file system, the latest is what is used for reliability. The events are stored in chunks, then upon a certain period of time it flush the chunks to the destination. If a chunk failed, it will retry later.

You can read more about buffering in the official documentation:

http://docs.fluentd.org/articles/buffer-plugin-overview