Docker daemon log taking up too much disk space

Question

Preamble

For clarification, this concern is for the docker daemon logs, not for docker's container-level logs. The log-driver and log-opts docker daemon configurations seem to handle container-level logs just fine.

Description

The issue we're running into is that the docker daemon log (located at /var/log/docker.err.log) log file is growing too quickly, and not rotating on our docker swarm production servers. We are naturally worried about disk space over the long term. Concretely, we've been running for almost a month now, and on one node, that log file has already grown to 5.1GB. As our VM's are expected to run for months on end, this is obviously a concerning trend.

The linuxkit OS that we're using has very similar configuration to the example docker configuration, with some additional configuration in the /etc/docker/daemon.json file, most notably that we set debug to True. That was clearly a mistake, as our tests show that, had we not used that option, our log files would be substantially smaller.

However, even if we had left the log-level at it's default of info, it looks to me like the log file could still cause issues if the server is left running for too long. One of my coworkers did some rough calculations, and his guess is that the file could still grow to something like 10GB if the daemon is left running for 6 months or so.

What We've Tried

We've been trying to get a manual logrotate solution working to protect against this, but it seems like the docker daemon never reloads it's log file, which means that when logrotate does it's thing and creates a blank new docker.err.log, the docker daemon continues to write at whatever offset it left off at, and backfills the rest of the file with null bytes, taking up as much space as it was before. We've tried some solutions involving sending the HUP signal to the docker process without any success; it seems like the docker daemon doesn't handle that signal, or at least not in a way that reloads it's log file.

The Question(s)

Is there an accepted way to implement rotating of the docker daemon logs?

It seems unusual that we can't find any information about this, as it seems likely somebody somewhere has bumped into this issue before. Or do others running on swarm periodically restart servers at some point? Ideally we'd love to find a linuxkit-based way of rotating that log file specifically

Additionally, is there a way to truncate the existing docker.err.log on a running server without shutting down the server or docker daemon instance? We'd like to avoid at all costs having to deploy the updated OS image just to prevent the docker daemon log from using up all our disk space.

Steps to Reproduce

• Create a linuxkit OS ISO with a docker daemon service with debug: True in it's configuration file
• Using that ISO, run a docker swarm cluster for several days
• Observe the growth of the docker.err.log file

Linuxkit Config

kernel:
  image: linuxkit/kernel:4.15.5
  cmdline: "console=tty0 quiet console=ttyAMA0"
init:
  - linuxkit/init:6061875ba11fd9c563fda6234b103ed9997ff782
  - linuxkit/runc:52ecfdef1ae051e7fd5ac5f1d0b7dd859adff015
  - linuxkit/containerd:13f62c61f0465fb07766d88b317cabb960261cbb
  - linuxkit/ca-certificates:0a188e40108b6ece8c2aefdfaaad94acc84368ce
 ...

services:
   - name: docker
    image: docker:17.12.0-ce-dind
    capabilities:
     - all
    net: host
    mounts:
     - type: cgroup
       options: ["rw","nosuid","noexec","nodev","relatime"]
    binds:
     - /tiles:/tiles
     - /etc/resolv.conf:/etc/resolv.conf
     - /tmp/hosts:/etc/hosts
     - /root/.ssh:/root/.ssh
     - /var/lib/docker:/var/lib/docker
     - /lib/modules:/lib/modules
     - /etc/docker/daemon.json:/etc/docker/daemon.json
     - /persistent:/persistent
     - /application:/application
    command: ["/usr/local/bin/docker-init", "/usr/local/bin/dockerd"]

files:
  - path: etc/docker/daemon.json
    contents: |
        {
          "debug": true,
          "data-root": "/persistent/docker",
          "insecure-registries" : ["foobar-docker-registry.chip:5000"],
          "log-driver": "json-file",
          "log-opts": {
            "max-size": "100m",
            "max-file": "4"
          }
        }

Answer 1

Just to close this off, this is no longer an issue in linuxkit. They have since added logging support using memlogd, and their own log writer, which handles automatic rotation. This is the link to their logging documentation.

The yaml is configured like so:

init:
  # A circular buffer that captures logs from onboot and service-level containers
  - linuxkit/memlogd:v0.7
  ...
services:
  - name: write-and-rotate-logs
    image: foobar/logwrite
    command: ["/usr/bin/logwrite",
              "-log-dir", "/persistent/log/bespin",
              # Keep at most 25 files (Note: file numbers are 0-based)
              "-max-log-files", "25",
              # Max log file size set to 200MB (200 * 1024 * 1024 = 209,715,200 bytes)
              "-max-log-size", "209715200"]

This will capture all linuxkit service logs, including docker daemon logs, and write them to disk. It handles rotation as it writes.