2

If the max-file value is set to 2, two files are created as shown below.

11111-json.log
11111-json.log.1

But here, when the 11111-json.log file size is max-size, the contents of 11111-json.log are moved to 11111-json.log.1, and the size of 11111-json.log Becomes zero. /var/log/container At this point I lose the last log.

The log in the /var/log/container path eventually links to /var/lib/docker/containers/~, so if the file mentioned above works that way, the log will be lost.

How can't I be lost?

김태우
  • 1,033
  • 1
  • 12
  • 29
  • Correct. You need to something to pick up and store or forward the json logs. A lot of kubernetes setups use [fluentbit](https://fluentbit.io/) or [fluentd](https://www.fluentd.org/) or [filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html) – Matt Aug 19 '19 at 10:37
  • thank you for the reply. I have already configured fluentD as a daemonset. The log in `/var/log/container` is read, but loses data the moment the log file in `/var/lib/docker/containers/~/11111-json.log` changes to `11111-json.log-1`. – 김태우 Aug 19 '19 at 10:49
  • What log data are you missing? fluentd should watch the rotated (old) file for 5 seconds for anything to be flushed to it before reading from the (new) blank file. – Matt Aug 19 '19 at 11:11
  • The 11111.log file in `/var/log/container/` links the `/var/lib/docker/containers/~/11111-json.log` file. When the `/var/lib/docker/containers/~/11111-json.log` file reaches max-size and is renamed `/var/lib/docker/containers/~/11111-json.log-1`, the last logs that fluentD has not yet read are not in `/var/log/container/11111.log`. – 김태우 Aug 19 '19 at 11:29
  • 1
    Maybe increase [`rotate_wait`](https://docs.fluentd.org/input/tail#rotate_wait) in your tail config. see [this answer](https://stackoverflow.com/a/54520584/1318694) – Matt Aug 19 '19 at 11:34
  • thank you. I haven't tested it yet, 5 seconds before all of the tests should have finished. So it doesn't seem to work. – 김태우 Aug 19 '19 at 11:51

1 Answers1

3

According to your settings, all logs .log.1, .log.2 are stored in /var/lib/docker/containers/... and as per docker documentation you can change those settings for docker in daemon.json: 

 "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",

in /var/log/containers you can find link to the last created log file.

As per documentation for flunetd: You should consider using in_tail option:

in_tail is included in Fluentd's core. No additional installation process is required. When Fluentd is first configured with in_tail, it will start reading from the tail of that log, not the beginning. Once the log is rotated, Fluentd starts reading the new file from the beginning. It keeps track of the current inode number.

Please refer to the similar community post

Mark
  • 3,644
  • 6
  • 23