4

Rancher v2.5 logging uses the banzai cloud logging operator - see here.

The operator deploys and configures a Fluent Bit DaemonSet on every node to collect container and application logs from the node file system. Fluent Bit queries the Kubernetes API and enriches the logs with metadata about the pods, and transfers both the logs and the metadata to Fluentd. Fluentd receives, filters, and transfer logs to multiple outputs

enter image description here

I don't know much about Fluent Bit, but the documentation says

Fluent Bit is an open source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources, enrich them with filters and send them to multiple destinations.

Sounds quite similar to Fluentd (and other log forwarders like LogStash).

So what would be the reason/benefit of having both Fluent bit and Fluentd as part of the logging operator?

Ryan.Bartsch
  • 3,698
  • 1
  • 26
  • 52

1 Answers1

6

Performance and resource utilization. FluentBit is much more lightweight and therefore less expensive to run as a DaemonSet. Forwarding on to FluentD also makes sense, since FluentD has hundreds more plugins and is generally more flexible and configurable. The FluentBit docs themselves say:

Fluentd is a great option due to it flexibility and availability of plugins (more than 300 extensions!) but if the data collection will happen in an Embedded environment or an IoT device where the system capacity is restricted, Fluent Bit is the solution to use.

So what it looks like this operator is doing is using FluentBit to forward the logs in as lightweight a process as possible, then using FluentD to do the heavy processing/aggregating/shipping to destinations, the same way an app might use its own log forwarder to send to FluentD (for example over the HTTP input from somewhere outside of Kubernetes) and then have FluentD manage processing those logs and shipping them to destinations.

Zac Anger
  • 6,983
  • 2
  • 15
  • 42