4

I have been using logrotation for years and never thought too much of it being a problem until I came across a question on stackoverflow (https://stackoverflow.com/questions/1508734/disable-java-log-rotation/) where someone wants to disable log rotation.

To me with experience in having build server and even production servers cleaned up manually because logs are not rotated and discs are running out and suddenly machines come to a halt that all seems crazy, but it occurred to me that maybe it is not so obvious after all.

So what are the benefits of log rotation? And what are the drawbacks (e.g. more difficult to debug/analyze maybe)? What tools do you find useful for working with rotated log files? Splunk I assume, but what else?

Community
  • 1
Manfred Moser
  • 173
  • 1
  • 7

3 Answers3

7

I think the benefits of log rotation are clear:

  1. You get easily managed smaller log files instead of one huge log file.
  2. You don't run out of disk space suddenly if you configure it appropriately according to your capacity limits. (size option)
  3. Older log files can be compressed so that log files get even smaller in size, and thus saving more disk space. (compress option).
  4. You can rotate the files in a specific way / time. For example, each log file contains only information related to a specific day. This will make the search easier given that you know the date. When you don't know the date, you can just search all files or a subset of them. (daily, monthly, etc).
  5. You automatically get rid of very old files. For example, you can keep 30 files at max. (rotate 30).
  6. You can add the extension you like such as rotation date. (dateext).
  7. You can execute specific scripts before/after rotation. (prerotate, postrotate).

EDIT: I added more items to the list and included the options when applicable. For more details, man logrotate can be consulted.

Khaled
  • 36,533
  • 8
  • 72
  • 99
  • And for httpd server, you forgot to say that it is possible to move them on another disk for further use, e.g. statistics (webalizer is your friend), without having to restart the server. – Olivier Pons Dec 21 '10 at 09:58
  • Saving the logs in S3 would help you look at particular requests (easy grep) to check for things which are time based – Sairam Dec 21 '10 at 11:12
  • Writing into the same file which goes to few Gigs over the course of few days and that too multiple boxes is bad. You always want to split them up in chunks which are meaningful. Think about Google's servers which log few Gigs / Tera bytes every hour . – Sairam Dec 21 '10 at 11:14
0

Benefits:

  • if log files are big, you can compress or delete old data to avoid filling up the disk
  • if you know when something happened, it'll be quicker grep a single day's log file than a single perpetual log file

Drawbacks:

  • if you want to process the entire history, you'll have to specify multiple file names
  • some programs don't support rotation, and if you rotate it using a utility like logrotate, there are some edge cases to watch out for (e.g. you have to use the truncate option, which I think means you risk losing a tiny amount of data)
Mikel
  • 3,867
  • 2
  • 20
  • 16
0

If you think about what you use logs for, the benefits become even more clear. Leaving aside the obvious points mentioned by others and yourself. For me the big benefit is that I can control how the logs are fed into offline analysis tools.

I've got an assortment of home-grown scripts I use for analysing logs - but, AFAIK, splunk and the various webloog analysis tools are the only 'standard' tools available off the shelf. (not counting realtime log file analyzers like fail2ban).

symcbean
  • 21,009
  • 1
  • 31
  • 52