Reducing log rotation frequency for AWS EC2

Question

I have a java app running on elastic beanstalk, with the log rotation to S3 set up and working fine. However, I find that my catalina.out log file only exists for up to 15 minutes before rolling to a new file, which makes debugging problems on the same day more tedious (get gz from s3 and unzip, instead of just ssh-ing and reading the log file).

Is there any way to configure this through tomcat/log4j or the elastic-beanstalk console? I believe it's an AWS thing, because we had the same app deployed on Dotcloud and it kept a single log file per day.

Answer 1

As of Today, not really. The only thing that you can do is modify the file /etc/logrotate.conf.elasticbeanstalk in the EC2 instance where your app is running.

Change the size of the log and that should put more log info in one file:

/var/log/tomcat6/catalina.out /var/log/tomcat6/monitor_catalina.log /var/log/tomcat6/tail_catalina.log {
    size 20M <-- change to the file size that you want
    missingok
    rotate 2
    compress
    notifempty
    copytruncate
    dateext
    dateformat -%s
    lastaction
        /bin/chown tomcat:elasticbeanstalk /var/log/tomcat6/*gz; /bin/chmod 664 /var/log/tomcat6/*gz
    endscript
}

There's another post that has more info here:

Rotating S3 Logging using log4j with Elastic Beanstalk

Answer 2

Even though this question is old and resolved, I hope the following helps more people like me who struggled with the whole concept.

How log rotation actually works

I believe an explanation on how log rotation actually works might be more helpful, as the technical details will surely change over time and differ across platforms (in my examples I use php).

1. Elastic Beanstalk log rotations are done through logrotate and have configuration files inside the /etc directory in the form of logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.<service name>.conf. As you can tell from the path, hourly indicates that logs should be rotated hourly. Note that this file doesn't actually rotate the logs, it's just a set of logrotate log-specific options that get executed when the rotation is actually performed.
2. Rotation actually happen through cron, a service that allows you to schedule when to run certain scripts. In this case, if you check the /etc/cron.hourly directory you will see files like cron.logrotate.elasticbeanstalk.<service name>.conf (don't let the .conf extension fool you, it's really an executable, note the #!bin/sh at the top). When cron executes this script, it will run logrotate for the specified logrotate conf file.
3. The file that actually dictates at what time cron runs hourly jobs is /etc/cron.d/0hourly. The file that dictates at what time it runs daily/weekly/monthly jobs is /etc/anacrontab. As a side note, I believe that all logrotate conf files inside the /etc/logrotate.d directory get executed daily, as specified in /etc/cron.daily/logrotate.

Your options

With this basic info, you should be able to understand how to change the frequency of rotation to your liking. For instance:

• If you just care about the interval and not the specific rotation time, you can simply move the rotation script to the appropriate /etc/cron.<interval> directory:

  mv /etc/cron.hourly/cron.logrotate.elasticbeanstalk.healthd.conf /etc/cron.daily/
  

• If you want more control, pull the script out of the cron directory so it stops getting executed at default intervals:

  mv /etc/cron.hourly/cron.logrotate.elasticbeanstalk.healthd.conf /etc/
  

  Then, create a custom cron job inside the /etc/cron.d directory, specifying when to run the script. Example:

  SHELL=/bin/bash
  PATH=/sbin:/bin:/usr/sbin:/usr/bin
  MAILTO=""
  HOME=/
  05 3 * * * root /etc/cron.logrotate.elasticbeanstalk.healthd.conf
  

  which will run once a day, every day at 3:05am.

  Note: if you decide to create your own script, make sure it has execute permission, otherwise cron won't be able to run it.

For me, modifying these files only worked as a predeploy platform hook. However, if you're not changing the default AWS log rotations/cron jobs but just adding your own, you can safely do that in a config file inside .ebextensions.