1

I am trying to understand the interaction between Docker and Fluentd in a K8s cluster. I have seen places where you need to configure Docker to output to a logging driver, and Fluentd can be used as logging driver, like here.

On the other hand, I have seen posts (like this or this) where Docker does not know the existence of Fluentd as a DaemonSet.

My whole intention is to do log rotation, however I am not sure if having Fluentd in place will actually rotate the logs Docker writes on, so I do not end up with the whole storage space in the node taken up by the logs over time. Is it enough to use FluentD DaemonSet without Docker knowing the existence of Fluentd?, o I need to somehow connect Docker to Fluentd with a driver as well?

FCR
  • 1,103
  • 10
  • 25

1 Answers1

3

Per official k8s logging architecture docker (or any other runtime) does not need to know about FluentBit, Fluentd, Filebeat, or any other log collector you use. In fact, you can use multiple log collectors a time!

enter image description here

The same document states that k8s is not responsible for log rotation, so you set up a logrotate yourself. Fluentd/FluentBit daemon on the other end also does not rotate log files, but it does able to track log rotation and adjust the tail cursor accordingly (by default).

By far the easiest way to implement the architecture is

Max Lobur
  • 5,662
  • 22
  • 35
  • "Ensure there's logrotate: many k8s worker AMIs, e.g. EKS already have it." Can you elaborate on this? This is the key part. On the other hand, you can also configure log-rotation with fluentd. How does that configuration relate to this solution you are proposing? https://docs.fluentd.org/deployment/logging#log-rotation-setting – FCR Sep 17 '20 at 11:23
  • `By default, Fluentd does not rotate log files.` - this is what I rely on all the time. 1. Sometimes it's system https://github.com/awslabs/amazon-eks-ami/blob/8a1b744a575d88f8137a823f860850838db6ac95/scripts/install-worker.sh#L133 2. Sometimes it's docker setting https://github.com/kubernetes/kops/blob/083e29e510c20cc92dfe22a5d7e118f24cca3e43/nodeup/pkg/model/docker_test.go#L64 . But I've never seen a k8s worker distro without built-in log rotation. – Max Lobur Sep 17 '20 at 11:36
  • If you building your own - yes you need to use any of those ways to rotate yourself. Do not use fluentd daemonset for this - worker health should not depend on objects created thru the API (e.g. daemonsets). If the daemonset rollout will be stuck somehow - you don't want your workers to go down. – Max Lobur Sep 17 '20 at 11:36
  • "Ensure" - check if it's there, if not - add it. – Max Lobur Sep 17 '20 at 11:36
  • I am planning to use Digital Ocean managed kubernetes, no access via ssh to the Nodes to activate logrotate, any idea how to perform this? – FCR Sep 17 '20 at 11:42
  • It's there 100%. This is a packaged product, that's why you don't have access. If there's any issues with managed nodes - poke their support, this is not your headache. – Max Lobur Sep 17 '20 at 11:44
  • 1
    Well, if this is true https://www.digitalocean.com/community/questions/is-logrotate-configured-for-k8s-clusters-by-do-or-do-i-need-to-configure-it-via-daemonset-myself - it sucks. On the other post, they mentioned logrotate: https://www.digitalocean.com/community/tutorials/modernizing-applications-for-kubernetes . You should be either able to fork worker image (use unmanaged), or they should solve all the system-level issues there (managed). – Max Lobur Sep 17 '20 at 11:48
  • @FCR how did it go? I plan to use DO too and found this https://www.digitalocean.com/docs/kubernetes/#log-rotation – Max Lobur Sep 25 '20 at 10:00