tail -F still stops on logrotate sometimes

Question

Linux ip-10-10-64-122 4.1.7-15.23.amzn1.x86_64 #1 SMP Mon Sep 14
23:20:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I'm using tail -F in a bash script which in turn loops over the results. This script is supposed to run indefinitely

tail -F /tmp/my.log |
grep --line-buffered "xyz" |
while read i
do
    echo "$i"
done
echo "end of read"

however while sometimes it works through a logrotate, other times it does not:

tail: '/tmp/my.log' has become inaccessible: No such file or directory

In this case I do not see "end of read" meaning the tail/while is still running/stuck. I can confirm the log file does get put back in place after logrotate within milliseconds as you would expect.

Logrotate config:

/tmp/mosquitto.log {
            rotate 500
            compress
            maxsize 40M
            nocreate
            missingok
            postrotate
                    /usr/bin/killall -HUP mosquitto
            endscript
    }

How can I force this to continue to tail?

Instead of providing the script, that runs the `tail` command, you need to provide the logrotate configuration for that log file. — ku4eto, Feb 04 '21 at 16:31

score 1 · Accepted Answer · answered Feb 05 '21 at 09:32

1

As i suspected, your log rotation basically renames the current log file, and then creates a new one. You need to use the copytruncate logrotate option, which keeps the original log file, along with all of its properties. This will allow tail to follow it properly. tail cannot follow a file, that does not exist anymore (the case when copytruncate is not used).

copytruncate: Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place.

answered Feb 05 '21 at 09:32

ku4eto

143
1
5

-F (capitol) is supposed to "retry" meaning it can follow a file that doesn't exist anymore, as long as it is created again soon. in-fact I have another "tail" running side by side using -F which continues through this error/rotate. – pstanton Feb 07 '21 at 00:57
https://man7.org/linux/man-pages/man1/tail.1.html – pstanton Feb 07 '21 at 00:58
While this is correct, the log rotation is not instant, and the tail may fail if it hits such a moment. The logrotation you are using, The tail script you are using may not be best way to do whatever you are doing. You may try to just grep the entire log file, instead of piping the tail output into it. Anyway, nothing stops you from changing the log rotation to use copytruncate. – ku4eto Feb 07 '21 at 09:46
i can't actually test this due to another constraint so will just accept. – pstanton Feb 07 '21 at 10:41

tail -F still stops on logrotate sometimes

1 Answers1