rsyslog stale file handler with catalina.out

Question

Problem: After deploying a microservices as a war via AWS EBS Tomcat 7 container...noticed that the log rotation which occurs at UTC day boundary leaves a stale inode file.

The log rotation is more of a copy n truncate, which causes a stale file handler for rsyslog, which is listening for changes to catalina.out. What's the best way to prevent stale inode descriptors? Should I specify a rollover policy in logback.xml or logrotate or ...?

output of sudo lsof /var/log/tomcat7/catalina.out (and sudo stat report latest inode)

rsyslogd 18970 root 2r REG 202,1 1250 134754 /var/log/tomcat7/catalina.out

but doesn't match log output of rsyslog in debug mode.

4638.114765354:7fc839b8c700: stream checking for file change on '/var/log/tomcat7/catalina.out', inode 135952/135952file 7 read 0 bytes

Workaround Stop Tomcat, remove catalina.out, then restart tomcat. This allowed rsyslog to continue streaming of new records.

However, after a few hours, rsyslog fails to stream newer log records to rsyslog destination server. The debug log of rsyslog contains the same inode as output of stat and lsof. If you run

sudo stat /var/log/tomcat7/catalina.out

rsyslog starts streaming again.

Have you noticed rsyslog stop streaming intermittently outside of the log rollover use case?

Why would a sudo stat /var/log/tomcat7/catalina.out cause rsyslog to stream again?

answering my own question: 1. appears that copyntruncate is occuring for catalina.out not just during UTC day rollover but throughout the day. 2. sudo stat tells the shell to reevaluate the inode location of whatever destination you give it...hence the reason why rsyslog client begins streaming again via imfile — babalu, Oct 23 '14 at 16:51

score 0 · Answer 1 · answered Aug 13 '15 at 13:35

I also have a problem with rsyslog not sending new records as soon as logrotated does its rotation on catalina.out. issuing a stat doesn't quite cut it for me as the problem is tomcat stops writing to catalina.out (!) ...after perusing various forums and blogs, I was able to address this through the following steps:

make sure that we have $WorkDirectory defined in rsyslog configuration; this allows rsyslog write the "state file" for catalina.out (or for any other log file(s) it watches, for that matter)
As noted here on Loggly's blog, you need to stop rsyslog, delete this state file, then restart rsyslog on a postrotate entry.

my logrotate setting for catalina.out (state files are in /var/lib/rsyslog):

/opt/tomcat/logs/catalina.out {
    rotate 7
    size 50M
    notifempty
    missingok
    postrotate
        service rsyslog stop
        rm /var/lib/rsyslog/*
        service rsyslog start
    endscript
}

rsyslog stale file handler with catalina.out

1 Answers1