How long do you keep log files?

Question

I have an application which writes its log files in a special folder. Now I'd like to add a functionality to delete these logs after a defined period of time automatically. But how long should I keep the log files? What are "good" default values (7 or 180 days)? Or do you prefer other criteria (e.g. max. used disk space)?

how long do you want to be able to check to see if there was a problem? — BlackICE, Apr 23 '10 at 19:40
Do you have a good reason not to use the logging system of the OS, so you can let the end user decide for themselves, centralize logging via Splunk, etc.? — hurfdurf, Apr 23 '10 at 20:42

score 9 · Accepted Answer · answered Apr 23 '10 at 20:01

There's no single answer. Factors to consider:

legal requirements for retention or destruction
company policies for retention and destruction
how long the logs are useful
what questions you're hoping to answer from the logs
how much space they take up

I've got some logs that roll over in days, and others that are kept for years.

I tend to keep access logs longer (in some cases, forever, once I've stripped identifying information out of them) than error logs as in theory, I don't need them once I've fixed the problems.

I keep some other debugging logs (showing user activity, how they're interacting with the system, etc.) around so we can see how things change with each version.

...

And this reminds me of anecdote when I worked for a university -- management brought in an outside consultant to do a third-party review of the webserver we were building. I was told to hand over 3 months of logs for them to review if we had sized the system appropriately. I knew that was a sign they didn't know what the hell they were doing, as universities are cyclic -- webserver load trended up over time, but with spikes at the beginning of each year, finals time, etc.

Do you separate the different type of log files? So do you have different log files for errors, debugging informations and user activity? — , Apr 23 '10 at 20:09
Yes ... it'd suck if I'd have to strip lines out of logs for retention. I guess it's nice to be able to have stuff together in one file when debugging, but I can grep by user identifier / IP address when debugging, so I can correlate user activity to the errors. If nothing else, keeping similar info in each file means they compress better. — Joe H., Apr 23 '10 at 20:45

score 5 · Answer 2 · answered Apr 23 '10 at 19:43

5

Depends on the requirements... there can be legal requirements for record keeping, and also you have to factor in how long they will be helpful for troubleshooting.

answered Apr 23 '10 at 19:43

David Stratton

453
2
10

score 1 · Answer 3 · answered Apr 23 '10 at 19:54

In addition to legal requirements and utility considerations, it depends quite a bit on how much disk space your logs are chewing up. I've seen very verbose programs rotate their logs every day, while very quiet programs rotate almost never.

Depending on your system, it might be worth considering the "logrotate" package instead of handling log rotation internally. Just place a file with reasonable defaults in /etc/logrotate.d/ and let your users modify it if they deem necessary.

score 0 · Answer 4 · answered Apr 23 '10 at 19:45

It depends on the industry that will be using your software. There are lots of production applications that will fall under government regulatory requirements that may require any outputs, including log files, to be kept for a specific period of time.

If your application falls into one of these areas or you are unsure, you should consult your companies legal department.

score 0 · Answer 5 · answered Apr 23 '10 at 19:48

You should for sure archive your logs before you run out of space. :)

In our organization, logs are archived daily. So the special "logging" folder will have only logs for current day. The archive is retained for 3 weeks on disk on a special archive location with lots of storage. This will help prod. support devs to go look if something gets reported. This is backed up to tape(yes tapes!) after 3 weeks. As a policy tapes are retained for another 35 days.

score 0 · Answer 6 · answered Apr 23 '10 at 20:03

I'm running into this same decision currently. I have an app that writes non-critical exceptions to a log file and what I've settled on is setting a configurable size threshold for the log file itself. Once the log file reaches a certain size it will be archived. The archive will be cleaned once every week. I'm also thinking of setting a "Growth rate" threshold so if the log fills up too quickly customer support will be notified. Not sure yet if that's overkill.

These logs are for support only so, in my case, there is no legal or business requirement to keep them past a week.

Do you have one big log file only? I create one file per day and log all types of informations (errors and debug informations). The size of the file depends on the current usage (I think max. 1MB per day). But I can imagine to increase log informations in the future. What threshold size did you choose and why? — Alex, Apr 23 '10 at 20:30

score 0 · Answer 7 · answered Apr 24 '10 at 12:24

The retention of log files depends on the criticality of the data being logged and the actual size of the storage medium on which the logs are stored as well as the various compliance procedures in the geographical location where the Server is hosted.

There is no hard and fast rule to the number of days the logs files should be retained though logs for at least a month ( space willing) would not be a bad idea.

Storing old logs as tar.gz files is also a nice idea if space is a constraint.

How long do you keep log files?

7 Answers7