Logcheck Ignore Logrotate

Question

I inherited a system using both logcheck and logrotate. The issue I'm having is that logrotate is sending a nasty e-mail saying something along the lines of:

*** WARNING ***: Log file (X) is smaller than last time checked!

Whenever logrotate rotates a watched file. It seems these tools would be designed to play nicely together, so I'm curious how to make it happen.

This seems simple and I apologize if I am asking a dumb question that is in the help files somewhere.

Ubuntu 18.04, logcheck 1.3.17, logrotate 3.11.0 - Copyright (C) 1995-2001 Red Hat, Inc.

Additional question, since 'copytruncate' is the method being used to manage the file rotation:

The service having its logs rotated is a Django / Apache application. The service itself (and file) can't really be stopped for the rotation process, because it's a webserver that requires uptime. Can you point me in the right direction for getting started on rotating these files cleanly? This process has had logrotate and logcheck running together quite cleanly, but we recently upgraded to Python 3.1 and that may have caused the underlying issue.

Second addendum: This issue may have been caused by developers getting irritated at some conflicting filesize limits (Django limiting logs to 4M and logrotate rotating at 5M) and truncating the logs themselves. Knowing that logrotate + logcheck has a strong proven track record was extremely helpful.

Answer 1

I've been using logcheck+logrotate on Debian systems for 20 years, and currently use them on ~60 Debian version 8, 9, and 10 servers, and I have never seen that message from logcheck. So I did a little digging.

The message does not actually come from logcheck itself, but rather from logtail2 (or, originally, logtail), which is in a subsidiary package that logcheck brings in as a dependency. According to the logtail package description, the primary difference between the two versions is that logtail2 is better at handling files which have been rotated. Given this, I'd start by checking which logtail your logcheck is using:

$ grep logtail /usr/sbin/logcheck 
LOGTAIL="/usr/sbin/logtail2"
        || error "Could not run logtail or save output"

Verify on the first line that LOGTAIL is being set to logtail2, not logtail.

Assuming that it is using logtail2 (as it should), the "log file is smaller" warning is only issued when the file shrinks and the inode number remains the same - in other words, if the file has been rewritten in-place. This is sometimes done to deal with services which aren't smart enough to re-create their log files as needed, so the log gets rotated by copying it and then resetting the size to 0 instead of by renaming it and creating a new file. Because this is an unusual (and rather hacky, not to mention risky, as it creates a race condition which could allow log messages to be lost) way to handle log rotation, I'm going to guess that you have only a small number of log files that this warning comes up for.

The best solution for this problem would be to fix your rotation settings for the relevant logs so that they're handled the "normal" way instead of by overwriting the old log. To do this, grep your logrotate rules for any logs that use the copytruncate option, then determine what needs to be done to get those services to allow rotation without using copytruncate - the brute-force way to achieve this would be to stop the service in a prerotate script and restart it in postrotate, but there's usually a better way, such as sending it a specific signal to tell it to re-open its log files. (If you don't find anything mentioning copytruncate, the technique could also be implemented manually in a rotation script, which you'd have to look for manually instead of using grep.)

If that's not an option, you might be able to create ignore rules in the logcheck rule directory telling it to ignore those warning lines, but that's dependent on whether they're handled as standard anomalous log entries or if they bypass the ignore system.