1

I'm running the following EKS version:

kubelet --version
Kubernetes v1.22.12-eks-ba74326

I've set the following parameter as documented here.

  "containerLogMaxSize": "100 Mi"

I then restarted the kubelet:

systemctl daemon-reload
systemctl restart kubelet

But my logs are still rotating at 10 MB.

Any ideas for how to debug or fix this?

merlin2011
  • 273
  • 1
  • 12
  • I don't believe it tolerates spaces between the number and the unit; did you check the `journalctl -u kubelet` to see if it whined on startup? – mdaniel Mar 29 '23 at 04:47
  • @mdaniel I do not see any complaining but I do see `FLAG: --container-log-max-size="10Mi"`, so the option is certainly being ignored or overridden somehow. I also tried removing the space between the number and the unit, but that did not do anything different. – merlin2011 Mar 29 '23 at 18:54
  • Hmm, ok, then the next thing I'd check is (as you said) see if EKS's scripts are doing something underhanded; start with `systemctl cat kubelet.service` and chase all the files and environment mentioned. You may have some luck with `grep -r container-log-max /etc` but that may not be the whole story since systemd is a complicated thing – mdaniel Mar 30 '23 at 02:49
  • 1
    My current suspicion is that it's this issue, so I'm going to try changing the runtime and see if that helps: https://github.com/kubernetes/kubernetes/issues/106855#issuecomment-990759441 – merlin2011 Mar 30 '23 at 23:24
  • @merlin2011 as you have mentioned above, can you please confirm if changing the runtime has resolved your issue. – Kiran Kotturi Apr 03 '23 at 12:55
  • @KiranKotturi It did not resolve the issue. :( I'm still investigating. – merlin2011 Apr 04 '23 at 19:12

2 Answers2

0

As per the official documentation

ContainerLogMaxSize is a quantity defining the maximum size of the container log file before it is rotated. For example: "5Mi" or "256Ki". If DynamicKubeletConfig is on, when dynamically updating this field, consider that it may trigger log rotation. Default: "10Mi"

If the value mentioned in the containerLogMaxSize is not accepted then by default the log rotation is returning 10 MB of log file.Please update the value as mentioned in the above example as 100Mi (without spaces) as your requirement is 100 MB.

containerLogMaxSize: 100Mi
Kiran Kotturi
  • 211
  • 4
0

I ran into this same issue and after some digging fixed it with the following.

Systemd was using the dropin file /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf for kubelet startup and this file pointed to a config file /var/lib/kubelet/config.yaml, this is where I made my changes.

$ sudo vi /var/lib/kubelet/config.yaml 
  added the following to the bottom of the file:  
    containerLogMaxFiles: 6  
    containerLogMaxSize: "50Mi"  

Then restarted the kubelet:
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet

Finally I used the following to check the settings, I ran this on the
kubernetes control plane server. Make sure to change NODE_SERVER_NAME to
your node server name:
$ kubectl get --raw "/api/v1/nodes/NODE_SERVER_NAME/proxy/configz"

Easier to read if you have jq installed
$ kubectl get --raw "/api/v1/nodes/NODE_SERVER_NAME/proxy/configz" | jq .
jbert
  • 1