We know logs are important. What is the best practice to maintain them?
Logs can get big - so we go for log rotation. And sure we don't want to lose the logs - so we go for their backups.
What is the best practice of handling these things? please advise.
Logrotate is a good tool for log rotation and compression. It is available for most Unix flavors. I find the defaults used on Ubuntu to be a good starting point. When changing rotation frequency, you should change the rotate count.
Keep logs as long as they may be useful. Archive to non-disk storage if necessary. For business systems there may be legal requirements directing retention and/or destruction of log data.
Depending on the data and availability of data in another log data may be kept for a week, a month, a quarter, or a year. Only duplicated data is deleted after a week. A monthly backup retained for a year will give you a years worth of most log data.
A centralized log server generally serves a different function than log retention. Comparing the centralized logs to host logs may detect log file tampering.
I think BillThor provides a good answer. As he said, centralized log collection is a good idea. If you put your logs into a database you can do all sorts of on the fly queries, which can be tremendously helpful. One useful open source log management tool is logstash. Commerical tools such as splunk serve the same purpose and might be appropriate if you want more formal support.
Something else to consider about managing logs on individual servers: log rotation has two components: rotating on a schedule, and rotating based on file size. Your log rotation should take both in to account. If you are just rotating a log file on a weekly basis and something suddenly starts logging huge amounts of data, you are going to run out of disk space. Similarly, if you are just rotating on file size you could quickly lose old logs if the log volume dramatically increases. This is another excellent reason to consider centralized log management.
