0

Forgive me if I am vague, but I am trying to keep this as secure as possible.

I have a CentOS server setup as a central remote logging server. Servers have been sending their logs to this server for a few months now, with no issue. These log files are rolled on a monthly basis, and have been working without incident. This month however, (without any configuration changes that I know of) rsyslog as decided to send new log files to the rolled log file (access_log-20150901), rather then the unrolled file (access_log).

The weird thing is that I even moved the rolled log (access_log-20150901) into an archives directory, and rsyslog is still sending updates to that file.

I am using custom filters, that look like this:

if $programname contains 'access_log' then /var/log/remotelog/access_log

However it is updating the file at:

/var/log/remotelog/archive/access_log-20150901

Anyone know what is going on here?

FYI. The permissions are all the same for all of these files and folders.

Edit: Fixed logrotate, with postrotate method to restart rsyslog after the logs have rotated

/var/log/remotelog/access_log
{
    missingok
    notifempty
    monthly
    create 0660 <user> <group>
    rotate 12
    postrotate
        /sbin/service rsyslog reload > /dev/null 2>/dev/null || true
    endscript
}

rsyslog version: rsyslogd 8.9.0.ad1

David Allen
  • 145
  • 1
  • 7
  • How is the log rotation (aka rolling) being performed? Are you using `logrotate`? It sounds like `rsyslog` wasn't HUP'd or restarted. – Gene Sep 03 '15 at 19:30
  • Whole rsyslog config will be really helpful for troubleshooting. Anyway, please could you post at least a single (sanitized) row ending to the wrong (as per your question) logfile? Also which version of rsyslog are you running? – Damiano Verzulli Sep 03 '15 at 19:35
  • The rsyslog.conf file is default, the only line of importance is "$IncludeConfig /etc/rsyslog.d/*.conf", which is where the custom filters are, which I included above. – David Allen Sep 03 '15 at 19:39
  • Also... based on "if" detective you posted and related logfile-path, I bet that rsyslog is NOT doing any rotation. I think "logtotate" is in charge, for this, in your set-up. Hence, it seems logtotate rotated logfile, without telling this to rsyslog, and rsyslog still write the the opened/OLD filehandle. Please add comments on this. – Damiano Verzulli Sep 03 '15 at 19:39
  • Okay, so do I have to restart rsyslog after the logs rotate? This has been working for the last few months without problem, and I don't think that anyone has been restarting rsyslog manually each month. But we definitely are using logrotate to handle log rotating. – David Allen Sep 03 '15 at 19:42
  • restarting rsyslog did the trick. Should I just put in a crontab to restart rsyslog once a month, or is there a better way to do that after the logrotate finishes? – David Allen Sep 03 '15 at 19:53

1 Answers1

1

You have two basic options here, IMHO.

1) add a "postrotate" clause to the logrotate stanza to issue a HUP to the rsyslog pid

2) add this file to the rsyslog filespec in the /etc/logrotate.d/rsyslog file.

The problem is that you never restarted the rsyslog daemon after rotating, so it's still holding on to the old filehandle internally. It probably worked fine up until now because the rotate of the rsyslog files took longer than the rotate of this file, which means the HUP on the rsyslog files came after this file had been rotated. Recently, this rotate took longer than the rsyslog files, so the HUP happened before this log file was closed/rotated. After the HUP, rsyslog still had the old file opened, and the rotate never notified rsyslog to update it's internal filehandle.

John
  • 9,070
  • 1
  • 29
  • 34