1

I have an ec2 c3.medium instance running. It has recently started maxing out the cpu making the server unresponsive. Rebooting the instance doesn't really help for long, although it does make apache serve sites again for an hour or so.

When I look at top I can see various apache processes running, this fluctuates between about 10 and 1, and the CPU% usage of these fluctuate up to about 30%.

When it is maxing out it is these apache processes that are the problem/

How can I find out what is driving these processes?

snapshot of top summary here

top - 16:06:32 up 33 min,  2 users,  load average: 1.80, 1.67, 2.13
Tasks:  87 total,   2 running,  85 sleeping,   0 stopped,   0 zombie
Cpu(s): 55.4%us,  2.6%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 42.0%st
Mem:   3842540k total,  1698436k used,  2144104k free,    87556k buffers
Swap:  1048572k total,        0k used,  1048572k free,   501328k cached




 2088 apache    20   0  707m  75m  31m S 17.9  2.0   0:53.59 /usr/sbin/httpd
 2188 apache    20   0  698m  70m  27m S 17.9  1.9   0:59.35 /usr/sbin/httpd
 1613 mysql     20   0  885m 200m 6860 S 10.6  5.4   3:41.55 /usr/libexec/mysqld
 2197 apache    20   0  698m  86m  44m S  5.6  2.3   0:53.89 /usr/sbin/httpd
 1864 apache    20   0  698m  78m  35m S  1.7  2.1   0:55.13 /usr/sbin/httpd
 2097 apache    20   0  699m  87m  43m S  1.7  2.3   0:54.70 /usr/sbin/httpd

Not ever had this kind of issue so beyond looking at top not sure how to pin down where the issue is coming from.

thanks.

UPDATE: Didn't hear back from my question in the comments but interestingly the problem just resolved itself after happening for about 2 weeks which suggests to me that AWS was stealing too much CPU rather than it being an issue with our setup. Anyone any comments on this conclusion?

Tofuwarrior
  • 239
  • 1
  • 3
  • 10

1 Answers1

2
  1. Load average > 1 != bad, you need to understand it better.
  2. That 42.0%st in the CPU line is not good. That is 'Steal Time' which means that the compute host you're on is likely oversubscribed and 42% of CPU cycles are happening in other instances, not yours.
  3. Run apachectl fullstatus and look for workers in the W state. Those will tell you what requests apache is serving at that moment.
Sammitch
  • 2,111
  • 1
  • 21
  • 35
  • thanks for your reply. I wasn't really focussed on the load average but that link will help me expand my understanding. – Tofuwarrior Jan 09 '15 at 09:20
  • re: steal time - the link you provided talks about Moving to a 'different physical server' to pin down if the issue is your host overselling the resource or you needing more CPU- in ec2 how can you be sure your new instance is ona different physical server? would you have to use a different region. – Tofuwarrior Jan 09 '15 at 09:42
  • re: load average. Having looked up my instance type it identifies Compute units: 3 (1 core x 3 unit) So how does this impact the load average assessment? Can I count this as 3 processors in whcih case my LA of 1-2 is 33-66% rather than 100%-200% – Tofuwarrior Jan 09 '15 at 09:55
  • I accepted your answer as you spent time helping me try to track down the issue. As noted above, looks like it might have been Amazon because problem seemed to just disappear. – Tofuwarrior May 09 '15 at 07:24