Monitoring AWS Systems Behind ElasticBeanStalk

Question

So I'm getting a company set up in the Amazon Cloud -- creating IAAS protocol/solutions/standardized implementation, etc while also being the SysAdmin for individual systems, app environments, and day-to-day uptime.

One of the biggest issues I'm having is tracking various system/application logs, as well as logging/monitoring/archiving system metrics like memory usage, cpu usage, etc etc In a centralized fashion. E.g. --> Nagios + Urchin.

The BIGGEST impediment to my endeavors is the following:

The company application is deployed in the form of a Java *.WAR file, uploaded to an Elastic BeanStalk application environment, load balancing and auto-scaling between 3(min) and 10(max) servers, and the EC2's that run the application are fired up and disposed of ad-hoc.

That is to say, I can't monitor the individual EC2's for very long because so many are being terminated then auto-provisioned/auto-scaled on the fly -- so I'd constantly be having to "monitor what I'm monitoring", and continuously remove/add EC2 machine addresses to my monitoring lists.

IS there some sort of way to use monitoring tools like Zabbix or Nagios to monitor the ElasticBeanStalk, and have it automatically add on new EC2's, and remove terminated/failed EC2's from its monitoring list automatically?

Furthermore, is there anything I can do with GrayLog to achieve similar results with the aggregation/centralization of my application logs from multiple EC2 instances into ONE consolidated set of logs/events? If not GrayLog, is there ANYTHING LIKE GrayLog that can automatically detect what EC2 members are being added/removed from the environment, and collect the logs from them automatically?

Any and all advice or direction is appreciated.

Thanks much, and cheers!!

Zenoss 4 will automatically follow your cloud instances as they get provisioned and terminated. Though I suspect that since this is something of a shopping question, it will get closed. — Michael Hampton, Oct 11 '12 at 21:02
I'm not really trying to shop, and I think I might unclear as to what you mean. I was trying to ask a technical question to my fellow technical geniuses, superiors, admins, etc...to see if anybody's got any suggestions, or tools, or administrative strategies I might be able to employ. Will look into Zenoss right away...thanks! — A. Avadis, Oct 11 '12 at 21:51
It seems to fall under the category of a "product, service or learning material recommendation." Please see the FAQ. :) — Michael Hampton, Oct 11 '12 at 21:59

bobmarksie · Answer 1 · 2016-03-25T17:15:12.247

If you are deploying a WAR on elastic beanstalk you can install the metrics by creating a configuration file in the .ebextensions folder under WEB-INF. See the following link for more information on configuring and instance using this: - http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers.html

To install disk / memory metrics you need to install the "Amazon CloudWatch Monitoring Scripts for Linux" - see http://aws.amazon.com/code/8720044071969977

files:
  "/opt/aws/cwms/CloudWatchMonitoringScripts.zip":
    mode: "000777"
    owner: ec2-user
    group: ec2-user
    source:  http://ec2-downloads.s3.amazonaws.com/cloudwatch-samples/CloudWatchMonitoringScripts-v1.1.0.zip
container_commands:
  01_unzip_cloud_watch_zip: 
    command: unzip -d /opt/aws/cwms /opt/aws/cwms/CloudWatchMonitoringScripts.zip
    ignoreErrors: true
  02_update_password_file:
    command: sed -i 's/Key=$/Key=<VALUE OF YOUR SECRET KEY>/;s/KeyId=$/KeyId=<VALUE OF YOUR ACCESS ID>/' /opt/aws/cwms/awscreds.conf
  03_update_crontab:    
    command: echo "*/1 * * * * /opt/aws/cwms/mon-put-instance-data.pl --mem-util --disk-path=/ --disk-space-util --from-cron" | crontab - -u ec2-user

Basically what this script does is download the Linux based CloudWatchMonitoringScripts.zip into a folder such as /opt/aws/cwms (this can be anywhere). The commands then unzip the file, update the access / secret key (using the "sed" command) and finally creating the crontab tab.

Be careful of the crontab tab section, as it could potentially wipe you existing crontab entries.

UPDATE (FEB 2016)

Here's an updated script which is working for me quite nicely as of Feb 2016 (see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-cw.html).

sources: 
  /opt/cloudwatch: http://ec2-downloads.s3.amazonaws.com/cloudwatch-samples/CloudWatchMonitoringScripts-v1.1.0.zip

commands:
  00-installpackages:
    command: yum install -y perl-Switch perl-Sys-Syslog perl-LWP-Protocol-https

container_commands:
  01-setupcron:
    command: |
      echo '* * * * * root perl /opt/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl `{"Fn::GetOptionSetting" : { "OptionName" : "CloudWatchMetrics", "DefaultValue" : "--mem-used --memory-units=megabytes --mem-util --disk-space-util --disk-space-used --disk-space-avail --disk-path=/" }}` >> /var/log/cwpump.log 2>&1' > /etc/cron.d/cwpump
  02-changeperm:
    command: chmod 644 /etc/cron.d/cwpump
  03-changeperm:
    command: chmod u+x /opt/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl

option_settings:
  "aws:autoscaling:launchconfiguration" :
    IamInstanceProfile : "MonitorRole"
  "aws:elasticbeanstalk:customoption" :
    CloudWatchMetrics : "--mem-used --memory-units=megabytes --mem-util --disk-space-util --disk-space-used --disk-space-avail --disk-path=/"

NOTE: You must have an IAM role called MonitorRule in place. It's role policy should be as follows (also see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-cw.html):-

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "cloudwatch:PutMetricData",
        "ec2:DescribeTags"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}

This answer doesn't work with the latest AMI, I needed to install a modules from perl following the instruction in: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/mon-scripts-perl.html The bad news for me, is that I tried to automatize this in app.config file, but is just not work: 00-installperldependencies: command: sudo yum install -y perl-Switch perl-Sys-Syslog perl-core perl-DateTime perl-LWP-Protocol-https — Jose Nobile, Mar 26 '15 at 22:00

score 1 · Answer 2 · answered Dec 29 '15 at 07:53

For Elastic beanstack's disk usage, you can enable the RootFileSysmtemUtil cloud watch instance metrics. This is available under Elastic beanstalk's Health configuration section. There is a "Health Reporting" section under "Health" configuration. Please select the RootFileSysmtemUtil option and save the configuration.

Now if you go to CloudWatch's ElasticBeanstack metrics, you can see the new metrics.

score 0 · Answer 3 · answered Oct 14 '12 at 09:16

We are using EC2, S3, etc but not yet ElasticBeanStalk. i can give you some suggestions and ideas...

Cloudwatch: We use CloudWatch from Amazon AWS which gives pretty good details about our EC2 instances. Monitoring setup is very simple and GUI stuff.. No scripting or anything needed. The Cloudwatch getting started guide will give lot of info about CLI, but developerGuide gives the exact info we will need: http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/AlarmThatSendsEmail.html http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/choosing_your_cloudwatch_interface.html

I found this link on Amazon ElasticBeanStack forum: - http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/using-features.healthstatus.html

Check this forum as well - "Amazon ElasticBeanStack forum" https://forums.aws.amazon.com/forum.jspa?forumID=86

Monitoring AWS Systems Behind ElasticBeanStalk

3 Answers3