0

So I have the following setup:

Server - t2.micro ECU Ubuntu instance Database - t2.micro RDS MySQL instance.

The stack is a basic LAMP stack with default Apache settings.

I am basically running a web crawler. I've setup some methods to run on the server based on a cronjob. I've noticed that after several minutes, the DB connections slowly rise to ~60, which I think might be too many. After 60, when I run: show processlist;

I get all the connections are command "Sleep", and the EC2 instance is unresponsive. I cannot ssh into it, and its pretty much unresponsive until I Stop/Start the instance, which changes the IP address and causes like 45 minutes of maintenance before its up again.

I know it probably has to do with the max number of connections, but I am confused why the EC2 instance crashes...? I monitor all the stuff on EC2 and RDS dashboards and nothing but 'DB connections' is going above the limit, EC2 seems like it doesn't even care, its at like 5% load on everything.

The application isn't logging any information, only access.log and errors.log are running and they don't seem to be full, is there something I haven't considered?

the tao
  • 9
  • 2
  • What is logged when the "crash" occurs? What is your definition of crash in this case? – EEAA Apr 19 '15 at 21:37
  • 1
    Additionally, these instances are good for nothing but the most light load, non critical workloads. You may want to try another instance type. – EEAA Apr 19 '15 at 21:38
  • Crash is when all my ssh clients time out at once, I am unable to ssh into the server. Shortly before I attempt to do 'sudo service apache2 stop' and get memory errors with bash. Which doesn't make sense because none of the metrics show memory is peaking... – the tao Apr 19 '15 at 21:40
  • Please post those memory metrics. – EEAA Apr 19 '15 at 21:41
  • I would have to replicate the problem but basically the metrics show very little activity in memory. – the tao Apr 19 '15 at 21:45
  • Well then there's nothing we can do to help. The more info you can provide, the better. – EEAA Apr 19 '15 at 21:46
  • I wouldn't use a t2.micro for anything but testing or accessing the EC2 command line tools. Their performance is quite sub-par and share heavily with other VM's. Use at least an m3.medium for production and MySQL workloads IMHO. – hookenz Apr 19 '15 at 22:01
  • Actually I realized there is no memory metric in EC2 in the monitor tab. I had to ssh into the server and use a little utility called htop to view memory utilization on Ubuntu. The memory was peaking out at 100% for quite some time, even after I cut the load in half lol. – the tao Apr 20 '15 at 00:30

1 Answers1

2

I suspect you may be running too high load for a t2 instance type. These instances are given cpu time based on a credit system, and t2.micros are have the least overall cpu capability in the instance class. These credits are accumulated at a rate of 6 per hour on a t2.micro, which gives you the ability to burst to 100% of the cpu core for 6 minutes. All other times you are only granted 10% of the core. Furthermore, you will only accumulate credits if your instance is completely idle.

I think you are using the available credits quickly (on both your web instance and your RDS instance), and are trying to run on too few cpu horsepower for the rest of the time, which is causing your instance to basically hang due to lack of resources.

If you want to spot-check my theory, you can look in the CloudWatch metrics for both your EC2 instance and your RDS instance, and look for the following: CPUCreditUsage and CPUCreditUnits. These graphs will show if you are accumulating and/or using your credits.

For further information, please refer to this page: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html

Your solution is likely to upsize either or both your instances to something larger within the T2 instance class, or to pick something outside the class entirely if you require sustained performance.

Chad Smith
  • 1,489
  • 8
  • 8