0

My main app server is experiencing unprecedented peak in network usage and load average.

The server is on Ec2/Scalr running php+memcache the graphs shows this kind of load has never happened before and that the load average builds up in hours see here: http://imm.io/56Jy I didnt change anything that could have this effect.

Additional info: PHP Version 5.2.4-2ubuntu5.10 Linux ec2-50-17-xx-xx.compute-1.amazonaws.com 2.6.16-xenU #1 SMP Mon May 28 03:41:49 SAST 2007 i686

How can I detect what is causing it?

Niro
  • 1,401
  • 4
  • 20
  • 36
  • have you run tcpdump to capture the source and type of network traffic? – sreimer Apr 20 '11 at 16:21
  • Did you checked apache logs, any anomaly? If you can stop apache and see if the server load drops. I would also check sendmail in case of misconfiguration there is an open relay. – Sameer Apr 20 '11 at 17:22
  • I would also check for a crawler. Get yourself a robots.txt file incase you don't have one because sometimes Google/MS crawlers can cause this kind of behaviour – Sameer Apr 20 '11 at 17:36
  • Thanks for the startpoints ! Re crawler- this is a web service with no in links and no content. its not a crawler. Re-tcpdump - going to try it! thanks for the tip. Re-Apache logs - going to check. I feel that some processes just hang and they keep accumulating quietly. since the load is going up slowly and steadily in the last 24 hours – Niro Apr 20 '11 at 18:32

1 Answers1

1

Turns out it was the AWS meltdown....

Niro
  • 1,401
  • 4
  • 20
  • 36