1

Our organization has been having a reoccurring problem at least once a week where our website gradually slows down until it stops responding to requests, can't SSH in, can't connect to websites, can't connect directly to MySQL database. This has been occurring for months and still haven't found the cause. We left the top command running on our webserver (CentOS 6) and connected over VMWare remote console once the server started slowing down to see if there was a specific process causing the issue (screenshot attached below).

The only other clue we noticed is that when the server starts slowing down, if we use the monitoring tools provided with VMWare ESXI we can see the disk read/write spike alot higher than average and stay really high. Only solution we've found is if we're quick enough before the server is going too slow, restart apache or mysql resolves the issue, or just rebooting the webserver. Anyone have any thoughts on possible causes?

top

  • side note: a monitoring tool like http://munin-monitoring.org/ can be useful – Fabian Apr 30 '18 at 16:10
  • Rather than relying on catching something at the right time in `top` ; please install one of the many more advanced [monitoring software packages](https://en.wikipedia.org/wiki/Comparison_of_network_monitoring_systems) , or something like new relic that will **record** trends and performance counters that will allow you to do a proper analyses – HBruijn Apr 30 '18 at 16:23
  • What does LSOF say during these periods? – TheFiddlerWins Apr 30 '18 at 17:18
  • @Fabian server has been up for a little longer than 24 hours and has yet to crash again (although I know it will crash again in a few days), here is a screenshot of my swap usage right now. https://i.imgur.com/Iywrp9V.png – mark-rodgers May 01 '18 at 23:32
  • @TheFiddlerWins I'll get back to you on this. Next time the server starts to slowdown I'll try to run lsof before it slows down to an unusable state. – mark-rodgers May 01 '18 at 23:34
  • @HBruijn know any free monitoring software that plays nicely with a LAMP stack running cPanel/WHM's Centos 6.8 distribution? – mark-rodgers May 01 '18 at 23:38
  • side note: if you want `top` to show "Swapped Size": run `top`, press f, scroll to SWAP, hit SPACE to toggle the selected line on/off, hit ESC. Press h for more details like sort, highlighting etc. Makes it a bit easier to see what is going on. (imho your "swap used" value is way to high) – Fabian May 02 '18 at 07:51

0 Answers0