3

Assuming I have set resource.limits.ephemeral-storage for containers in a Kubernetes cluster (using Docker), and the following Docker daemon.json logging configuration on the worker nodes:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10",
  }
}

My understanding is that all log files (even the rotated log files) will count towards this ephemeral storage limit. This means that to determine the value for resource.limits.ephemeral-storage, I have to consider the maximum allowed log size (here 10*100MB) to the calculation.

Is there a way to "exclude" log files from counting towards the container's ephemeral-storage limit?

Since log handling is done "outside" of Kubernetes, I want to avoid that the resource limits for Kubernetes workloads depend on the Docker log configuration. Otherwise any change to the rotation settings (e.g. increase to 10*200MB) could cause pods to be evicted, if one would forget to adjust the limit for each and every container.

hessfr
  • 33
  • 5
  • Hello @hessfr, welcome to StackOverflow! Please confirm how did you set up the cluster? Have you used Docker Desktop or some other solution? – Mikołaj Głodziak Jun 22 '21 at 12:04
  • Thanks @MikołajGłodziak! The cluster is set-up with Rancher RKE and is using Kubernetes v.1.17 – hessfr Jun 22 '21 at 14:23
  • You are probably using the deprecated version of Kubernetes. https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.5.8/ – Mikołaj Głodziak Jun 23 '21 at 06:50
  • I don't think the versions are a problem here: Rancher 2.5.5 supports k8s version 1.17.16 (as per https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.5.5/). My question was actually meant more general: if a newer version of k8s/docker would support "excluding" the rotated logs files from counting towards the ephemeral storage, I would be happen to upgrade – hessfr Jun 23 '21 at 12:49
  • what makes you think kubelet includes docker logs in pod disk usage calculations? At the moment, I have the impression that only ephemeral disk usage and container image size are considered. – Mikołaj Głodziak Jun 24 '21 at 13:47
  • You can verify it by creating a dummy log file larger than the ephemeral storage resource limit of the pod in /var/lib/docker/containers// (given that json-file is used as log driver). This will cause the pod to get evicted – hessfr Jun 25 '21 at 17:25

1 Answers1

1

Based on the function calcEphemeralStorage from release 1.17.16 source code, if you want to exclude logs from calculation you can comment or remove those lines and rebuild kubelet:

if podLogStats != nil {
        result.UsedBytes = addUsage(result.UsedBytes, podLogStats.UsedBytes)
        result.InodesUsed = addUsage(result.InodesUsed, podLogStats.InodesUsed)
        result.Time = maxUpdateTime(&result.Time, &podLogStats.Time)
    }

This part of the code is responsible for counting ephemeral storage usage for logs. But removing that part of code may also require to adjust some test files which expect that logs amount is calculated. All statistics are instead counted in this function:

func (p *criStatsProvider) makePodStorageStats(s *statsapi.PodStats, rootFsInfo *cadvisorapiv2.FsInfo) {
    podNs := s.PodRef.Namespace
    podName := s.PodRef.Name
    podUID := types.UID(s.PodRef.UID)
    vstats, found := p.resourceAnalyzer.GetPodVolumeStats(podUID)
    if !found {
        return
    }
    logStats, err := p.hostStatsProvider.getPodLogStats(podNs, podName, podUID, rootFsInfo)
    if err != nil {
        klog.ErrorS(err, "Unable to fetch pod log stats", "pod", klog.KRef(podNs, podName))
        // If people do in-place upgrade, there might be pods still using
        // the old log path. For those pods, no pod log stats is returned.
        // We should continue generating other stats in that case.
        // calcEphemeralStorage tolerants logStats == nil.
    }
    etcHostsStats, err := p.hostStatsProvider.getPodEtcHostsStats(podUID, rootFsInfo)
    if err != nil {
        klog.ErrorS(err, "Unable to fetch pod etc hosts stats", "pod", klog.KRef(podNs, podName))
    }
    ephemeralStats := make([]statsapi.VolumeStats, len(vstats.EphemeralVolumes))
    copy(ephemeralStats, vstats.EphemeralVolumes)
    s.VolumeStats = append(append([]statsapi.VolumeStats{}, vstats.EphemeralVolumes...), vstats.PersistentVolumes...)
    s.EphemeralStorage = calcEphemeralStorage(s.Containers, ephemeralStats, rootFsInfo, logStats, etcHostsStats, true)
}

In the last line you can find a usage of calcEphemeralStorage.

In the recent version the mentioned code include the same log calculation section, so the solution should work for the latest release too.

See also:

Mikołaj Głodziak
  • 4,775
  • 7
  • 28