How to make a cron script for cleaning up a log file?

Question

I don't usually do this kind of stuff, so I just have to ask. How do I make a script which checks a log file, lets say it's named log.txt, grabs everything but the first 100 000 lines from it to a new log.txt file and deletes the old log.txt? It would also have to run monthly.

So far with my zero coding knowledge I've only managed to make a script which would delete the log.txt once a month (new one generates automatically), but I would really like to keep data except the oldest 100 000 entries as mentioned below.

Here's the old script

#!/bin/sh
find /mnt/usb_storage/ -type f -name "log.txt" -exec rm -r {} \;

how I implemented it:

# cd /privRoot
# chmod 755 deleteLogMonthly.sh
# crontab -e 

59 23 1 * * deleteLogMonthly.sh

use logrotate for that, it does this. – Dennis Nolte Jul 10 '15 at 08:24 — Dennis Nolte, Jul 10 '15 at 08:24

score 4 · Answer 1 · edited Apr 13 '17 at 12:14

Your are attempting to reinvent wheels.

Logrotate does this efficiently:

https://www.linode.com/docs/uptime/logs/use-logrotate-to-manage-log-files

http://www.rackspace.com/knowledge_center/article/understanding-logrotate-utility

https://www.digitalocean.com/community/tutorials/how-to-configure-logging-and-log-rotation-in-apache-on-an-ubuntu-vps

Rather than a hard 100K lines, you give it a time interval and it will start writing to a new log file from that period, keeping the old one.

If, for some strange reason, logrotate isn't a true option, check out Cronolog:

Is there any alternate to logrotate for apache logs?

http://linux.die.net/man/1/cronolog

Or indeed manually craft a bash script like this one:

http://www.paperstreetonline.com/2014/12/09/bash-script-an-alternative-to-logrotate-d-to-rotate-asterisk-log-files/

That's still based on days, but you could add some line counting in there.

Hey, I am aware of Logrotate actually, but for some reason unknown to me, people here don't want to implement it, so I'm stuck reinventing wheels here. :) — Guy395, Jul 10 '15 at 08:40
That's strange. Before building that wheel, there are other similarly round wheels too: http://serverfault.com/questions/159450/is-there-any-alternate-to-logrotate-for-apache-logs & http://www.paperstreetonline.com/2014/12/09/bash-script-an-alternative-to-logrotate-d-to-rotate-asterisk-log-files/ (Also put in answer above) — JayMcTee, Jul 10 '15 at 08:49

score 1 · Answer 2 · edited Jul 10 '15 at 08:49

1

Add following in your logrotate configuration (new file under /etc/logrotate.d/)

/mnt/usb_storage/log.txt {
           rotate 5 # Change this to number of files you need to preserve
           monthly
           prerotate
               /usr/bin/sed -e '1,100000d' < /mnt/usb_storage/log.txt
           endscript
           postrotate
               <some another action>
           endscript
       }

you can verify it using logrotate -d /etc/logrotate.d/mylogrotate.conf (assuming you have created that file)

edited Jul 10 '15 at 08:49

Soviero

4,366
8
36
60

answered Jul 10 '15 at 08:47

Nehal Dattani

581
2
10

Okay, I could give this a try, where exactly in logrotate.conf do I add this? – Guy395 Jul 10 '15 at 09:03
Better create a new file, if you are just testing. You can create it anywhere and use it for testing purpose. just remember to give correct path when you are testing. Once you are sure that it works as per your requirement, put it under /etc/logrotate.d – Nehal Dattani Jul 10 '15 at 09:07
Alright, now I have mylogrotate.conf under loglogrotate.d all configured. Since my system only generates one log file, shouldn't rotate be 2 instead of 5? This is the only part that I don't get :) – Guy395 Jul 10 '15 at 10:07
1

From man page, rotate count Log files are rotated times before being removed or mailed to the address specified in a mail directive. If count is 0, old versions are removed rather then rotated. – Nehal Dattani Jul 10 '15 at 10:10
Alrights thanks, all of you guys have been extremely helpful, now one last question, for testing purposes, how do I make this rotate hourly? Apparently it isn't possible simply by editing myconf, since it gives me an error. – Guy395 Jul 10 '15 at 10:38
logrotate runs from cron, so change the crotab entry to logrotate more often – Dennis Nolte Jul 10 '15 at 11:01

score 0 · Answer 3 · answered Jul 10 '15 at 10:14

lookup the man page of "logrotate" or make it yourself by shell script. there is no magic to it. if you know how to make it by hand, just put it all together in a txt file and you have it.

also check out the man page of the tool "tail". it is designed to display the "tail" or more precisely the latest X lines of a file. "man tail" gives you more information about how to use it.

How to make a cron script for cleaning up a log file?

3 Answers3