shell script to delete 6 month old logs from log file

Question

I have log file of size 6.2 GB which is in use by the process and it keeps writing logs into this file. Now I don't want to rotate logs however i do want to clean this log file. I want to delete logs which are older than 6 months from this file. Is it possible using shell script to read the log file and delete the logs which are older than 6 months?

below is the log format

10.0.3.xx - - [17/Jun/2016:14:21:59 +0000] "GET /visible_topics HTTP/1.1" 200 581 "-" "help/97a1dd7eb981421b9719adde381560a78bed0b66 (ip-10-0-3-xxx; user1; 4411) ruby/2.3.0 (0; x86_64-linux)" 

Answer 1

The solution really is to use logrotate: it works very well, and it can also compress the rotated log.

If you really don't want to use logrotate, you had to manually cat/copy the relevant section of your log file into a new one, then removing the old one.

Without a detailed example of your log format it is not possible to give you any advice about the required shell command.

Anyway, you should really use logrotate.

Answer 2

I want to delete logs which are older than 6 months from this file.

I'm not clear what you mean but if I had to guess, you want to remove logfile entries from this file which are older than 6 months. If that's the case, it can be done but it's a rather extraordinary way of managing logs on a *nix system.

What you want to do (if my assumption is correct) is going to require parsing each line from that file and determining the timestamp in order to test if the entry date is within your current 6 month window. Depending on a pass/fail of the timestamp you would write out the new logfile without the "old" entries. You could use shell tools such as grep and head and tail to achieve this but in the interest of performance you may want to consider a custom compiled C application specifically tailored for what you need.

I don't know what kind of hardware this logfile is on but the sheer size of this logfile (6.8 Gigabytes) will likely cause significant performance bottlenecks when parsing the old entries and writing the new ones. In most cases when things are this much trouble it's usually a sign the process is in need of review. There are corner cases to that rule however. Good luck.