Why after delete some lines by sed, Postfix can't write maillog

Question

I want to use cron job, that once per three day will clean and sort maillog.

My job looks like

 /bin/sed -i /status=/!d /var/log/maillog | 
    (/bin/grep "status=bounced" /var/log/maillog | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" | /bin/sort -u >> /root/unsent.log) | 
    (/bin/grep "status=deferred" /var/log/maillog | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" | /bin/sort -u >> /root/deferred.log) | 
    (/bin/grep "status=sent" /var/log/maillog | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" | /bin/sort -u >> /root/sent.log) | 
/bin/sed -i "/status=/d" /var/log/maillog

Job works fine and do 3 step:

1. Delete from maillog all lines that don't contain "status="
2. Sort sent, bounced, deffered in different logs.
3. Delete from maillog all lines that contain "status"

After this job my maillog is fully clean and sorted to 3 logs.

But Postfix doesn't want to write next records to maillog.

I delete sed command, and Postfix writes next records fine.

Why sed command blocks maillog after execution cron job?

Answer 1

sed -i will unlink the file it modifies, so syslog/postfix will continue writing to a nonexistent file.

From http://en.wikipedia.org/wiki/Sed:

Note: "sed -i" overwrites the original file with a new one, breaking any links the original may have had

It is more common to process log files after rotating them out of place with a tool like logrotate or savelog, so that syslog can continue writing uninterrupted.

If you must edit /var/log/maillog in place, you can add a line to the end of your cron job to reload syslog when you are done. Note that you can lose log lines written to the file while your script is running if you do this. The command will depend on what distribution / operating system you are running. On ubuntu, which uses rsyslog, it would be reload rsyslog >/dev/null 2>&1.

Answer 2

I've reformatted your original code to highlight the pipe-lines you added

 /bin/sed -i /status=/!d /var/log/maillog \
 | (/bin/grep "status=bounced" /var/log/maillog \
     | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
     | /bin/sort -u >> /root/unsent.log\
   ) \
 | (/bin/grep "status=deferred" /var/log/maillog \
   | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
   | /bin/sort -u >> /root/deferred.log\
   ) \
 | (/bin/grep "status=sent" /var/log/maillog \
   | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
   | /bin/sort -u >> /root/sent.log \
   ) \
 | /bin/sed -i "/status=/d" /var/log/maillog

As @alberge noted, you could very likely lose log messages with all of this sed -i processing on the same file.

I propose a different approach:

I would move the maillog to a dated filename, (the assumption here is that Postfix, will create a new file with the standard name that it 'likes' to use (/var/log/maillog).

Then your real goal seems to be to extract various categories of messages to separately named files, i.e. unsent.log, deferred.log, sent.log AND then you're discarding any lines that don't contain the string status= (although you do that first).

Here's my alternate (please read the whole message, don't copy/paste/excute right away!).

 logDate=$(/bin/date +%Y%m%d.%H%M%S)
 /bin/mv /var/log/maillog /var/log/maillog.${logDate}

 /bin/grep "status=bounced" /var/log/maillog.${logDate} \
 | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
 | /bin/sort -u \
 >> /root/unsent.log.${logDate} 

 /bin/grep "status=deferred" /var/log/maillog.${logDate} \
 | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
 | /bin/sort -u \
 >> /root/deferred.log.${logDate}

/bin/grep "status=sent" \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u \
>> /root/sent.log.${logDate} 

To test that this code is working, replace the 2nd line ( /bin/mv .... ) with

  /bin/cp /var/log/maillog /var/log/maillog.${logDate}

Copy/paste that into a terminal window, confirm that the /var/log/maillog.${logDate} was copied correctly, then copy/paste each section, 1 at a time and check that the expected output is created in each of the /root logfiles.

(If you get error messages for any of these blocks, make sure there are NO space/tab chars after the last '\' char on each of the continued lines. OR you can fold each of those 3 pipelines back into one line, removing the '\' chars as you go.

(Note that to create each of the /root logfiles, I don't use any connecting sections via pipes surrounded by sub-processes. But, in other situations, I do use this sort of technique for advanced problems, so don't throw the technique away, just use it when it is really required ;-)!

After you confirm that all of this is working as you needed, then you extend the script to do a final cleaning up :

/bin/rm  /var/log/maillog.${logDate}

I've added ${logDate} to each of your output files, but as I see you're using sort -u >> you may want to remove that 'extension' to your sub-logfile names (unsent.log, deferred.log, sent.log) And just let those files get grow naturally. In either case, you'll have to comeback at some point and determine how far back you want to keep this data, and develop a plan and method for how you'll clean up these logfiles when they're not useful. I think someone mentioned logrotate package. You might want to look into that as your long-term solution.

---

This solution avoids a lot of extra processes being created, and it eliminates (mostly) the possibility of lost log records. I'm think you might lose all or part of a record if Postfix is writing to the logfile in the same split-second as you are moving the file. But your solution would have similar problems AND more opportunities for that to happen.

If I have misunderstood the intention of your design, using the nested ( .... ) | ( .... ) sub-processes, sorry! Consider updating your post to include why you are using that techinque.

I hope this helps.