3

I understand that copytruncate is not an ideal setup, but that's the cards I'm dealt with...

I have a service that is tailing data from journalctl and streaming it to a file.

The setup to rotate the file is...

/var/log/xxxxxx/xxxxxx.log {
    size 100M
    copytruncate
    rotate 5
    compress
    compresscmd /bin/xz
}

When I look at the location /var/log/xxxxxx/, I see that xxxxxx.log is 8GB but I also see xxxxxx.log-20181125.gz and so on... which means that rotation is happening.

The question I have is shouldn't the main xxxxxx.log be at 100MB why is it 8GB? And also if it's 8GB which cause the logrotate rule to fire, does this means that it's also constantly trying to process that 8GB file so essentially consuming resources for nothing?

UPDATE There is a service that runs: /bin/sh -c '/bin/journalctl --since="5 minutes ago" --no-tail --follow --unit="xxxxxx*.service" >> /var/log/xxxxxx/xxxxxx.log 2>&1'

When running logrotate -v /etc/logrotate.d/xxxxxx logrotate indicates that it truncated the file, but in fact it did not. I stopped the service and manually ran it again and it worked.

Running: CentOS Linux release 7.5.1804 (Core)

user432024
  • 283
  • 1
  • 5
  • 15
  • How often do you run logrotate, and how fast does the file grow ? 8 GB of logs per day will be a 8 GB file, if you call logrotate daily. – John Mahowald Nov 26 '18 at 19:58
  • It's in cron.daily. So I figured something. We have a service that runs: /bin/sh -c '/bin/journalctl --since="5 minutes ago" --no-tail --follow --unit="xxxxxx*.service" >> /var/log/xxxxxx/xxxxxx.log 2>&1' I even ran logrotate -v /etc/logrotate.d/xxxxxx and the file didn't truncate. Once I stopped the service and ran it again manually it truncated. – user432024 Nov 26 '18 at 20:48
  • "Copytruncate is not an ideal setup." That depends entirely on your use case, is cruft, and should be removed. "I stopped the service and manually ran it again and it worked." It's not magic. It's not specifically caused by a missing redirect. You are resetting the pointer used by the process. It has nothing to do with file size. – Chaim Eliyah Sep 25 '20 at 21:05

2 Answers2

2

Try running a tool like wc to see how much data is in the file. You may be getting a sparse file with the next record at the file position just after the position where the file was when it truncated and nothing before that. Some services keep their file position and write at that position. This does not work well when using copytruncate to rotate the log file.

Some applications reopen their files when sent a signal, and it is common to allow logrotate to move and recreate the file as log data will continue to be written to the log file until it is reopened. Use delaycompress when using this method. The logrotation specification for the syslog daemon is a good example of this.

See the man page for logrotate for a discussion on rotating logs.

BillThor
  • 27,737
  • 3
  • 37
  • 69
2

Found the issue... So I had setup my system according to this link: https://github.com/dcos/dcos-docs/blob/master/1.9/monitoring/logging/aggregating/elk.md

Notice here...

...
-u dcos-logrotate-master.service \
> /var/log/dcos/dcos.log 2>&1'
ExecStartPre=/usr/bin/journalctl

There's a > missing. Their docs have since been corrected. But since I did this step once when I first installed the issue was hidden until we noticed the logs where growing and not truncating...

user432024
  • 283
  • 1
  • 5
  • 15