Changing LogRotate's default way of rotating the log files

Question

My observation of logrotation is that to rotate any log file, the logrotate process, in the following order

1. copies the log file in question (lets call it file1.log) with a new name (adding time stamp or a number to the existing name so that it becomes file1.log-20140513),
2. deletes the existing file (file1.log) and creates a new blank log file with the original name (file1.log),
3. compresses the rotated file (file1.log-20140513), which creates a new compressed file (file1.log-20140513.gz), if compression option is set,
4. deletes the rotated file (file1.log-20140513), and the finally,
5. move to the next file to do the same above 4 steps to it.

I am having the following problem with this process:

1. My log files are huge in size (more than 10 Gb each) and I have around 42 such log files.
2. The processes which write to these files work in sync,
3. The DiskIO on the server is good, but still, copying takes time, and compressing also takes time and compression also consumes CPU.
4. I want all the newly created log files to have logs starting from same time.

To do that, I want logrotate to move, some thing which the mv command does, which renames the files instead of copying them. As for compression, I can disable that and trigger it via a different script scheduled via cron. But I want logrotate to move the files instead of copying them.

Now, am sure this is some thing the authors of logrotate too would have thought as it obviously saves disk IO and time taken to complete the entire logrotate operation, so I want to why files are being copied instead of being moved or renamed, and how do I achieve this via logrotate.

NOTE: I tried to do it manually, which was, move the file to which a running process was writing too, and create a new blank file with same name and same permissions (which is root, which is also the permission which the process is running with), but after moving and creating a new file, I saw that the process was not writing anything to it, so had to restart the process to make it write to that file. Can any one explain this behaviour too as to why logrotate manages to make the process write to the same file but I'm not able to using the simple steps.

Answer 1

The behavior you are describing only happens if logrotate has been explicitly told to do so through the copytruncate directive. The documentation warns about the possibility to lose some log data due to this behavior. That directive should only be used as a last resort.

The standard method of rotating logfiles is to rename and then send a signal to the process to let it open the new logfile. This is faster and doesn't risk losing part of the log. But it requires the writing process to be able to switch to a new logfile.

The compression can be turned off or postponed until the next rotation. If the compress directive is used, the old logfiles are compressed. If that directive is not used, they are not compressed.

If both compress and delaycompress are used, compression is delayed until the next rotation. That way after each rotation the two newest logfiles will not yet have been compressed.

Answer 2

after moving and creating a new file, I saw that the process was not writing anything to it, so had to restart the process to make it write to that file

Process is writing to the same file, thats why lograte copy instead of move. When you delete log, process still writes to log and you can see filesystem usage growing, but no file. Process restart will free disk space.

consider to

• write less into logs, is all logged information neccessary?
• read documentation, for example: http://www.thegeekstuff.com/2010/07/logrotate-examples/