0

I need to rotate logs files created by node processes as file sized keeps on increasing. Application runs in ubuntu on aws ec2 instances.

I have few forever node processes running which writes logs into below files.

forever.log
stderr.log
stdout.log

My CloudWatch Agent also looking at these files and pushes the logs to cloudwatch and node applications also continuously write logs into these files.

I need to find the best way to rotate these log files and clean up older files without affecting any of the above processes.

unsure below approach going cause troubles since it's going to move the file and going to create a new file

#!/bin/bash 
touch /script_logs/test.log
MaxFileSize=2048
while true
do
     sh test.sh >> /script_logs/test.log
#Get size in bytes** 
    file_size=`du -b /script_logs/test.log | tr -s '\t' ' ' | cut -d' ' -f1`
    if [ $file_size -gt $MaxFileSize ];then   
        timestamp=`date +%s`
        mv /script_logs/test.log /script_logs/test.log.$timestamp
        touch /script_logs/test.log
    fi

done

Need to implement a log rotation mechanism without affecting Cloudwatch agent which reads the log file or the application which write logs

Mahela Wickramasekara
  • 603
  • 1
  • 11
  • 22
  • 1
    write a config to `logrotate` and call it from cron or systemd.timer ? I believe nowadays better way would be to forward the logs to journalctl instance with some custom tags and let it handle all the rest. – KamilCuk Jul 13 '19 at 15:31
  • @KamilCuk Thanks for this a lot. I haven't had much experience with logrotate. Would you mind helping me out with a config that I could use with my scenario ? thanks for the help! – Mahela Wickramasekara Jul 13 '19 at 16:33
  • ? `size=2048` probably you don't care so `rotate 1000` and just the standard `compress` and `compressxz`. I thikn you also want `dateext` with `dateformat .%s`. Good to add `notifempty` and `missingok` if you want. Really, logrotate format is not that hard. – KamilCuk Jul 13 '19 at 17:16
  • @KamilCuk so for each log file path I can add configurations ? how do I add this to cron ? – Mahela Wickramasekara Jul 14 '19 at 01:04

0 Answers0