5

My team has a special requirement to delete all pod logs every X hours. This is cause the logs contain some sensitive info - we read and process them with fluentbit, but it's an issue that the logs are still there after. I couldn't find any normal way to rotate them by time, only recommendations on the docker daemon logging driver that rotates by file size. Is it possible to create a k8s cronjob to do something like "echo ''> /path/to/logfile" per pod/container? If yes, how?

I'd appreciate any help here. Thanks!

David Maze
  • 130,717
  • 29
  • 175
  • 215
J. Doe
  • 161
  • 2
  • 2
  • 5

2 Answers2

7

Kubernetes doesn’t provide built-in log rotation, but this functionality is available in many tools.

According to Kubernetes Logging Architecture:

An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node. Kubernetes is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that. For example, in Kubernetes clusters, deployed by the kube-up.sh script, there is a logrotate tool configured to run each hour. You can also set up a container runtime to rotate an application's logs automatically.

Below are some examples of how the log rotation can be implemented:

You can use them as a guide.

Wytrzymały Wiktor
  • 11,492
  • 5
  • 29
  • 37
1

Since kubernetes 1.21 you can configure the kubelet to rotate logs automatically. It is not done by time but by size, you may able to tune the size so fluentbit can pick the logs and after a short period kubelet rotates them.

If you configure rotation, the kubelet is responsible for rotating container logs and managing the logging directory structure. The kubelet sends this information to the container runtime (using CRI), and the runtime writes the container logs to the given location.

You can configure two kubelet configuration settings, containerLogMaxSize and containerLogMaxFiles, using the kubelet configuration file. These settings let you configure the maximum size for each log file and the maximum number of files allowed for each container respectively.

Full docs here

NicoKowe
  • 2,989
  • 2
  • 19
  • 26