0

My site (which as running fine until yesterday - CPU was at 2%, memory at 50%) keeps maxing out the resources - CPU and Memory at 100% and "load" and Disc IO very high.

the site is a php (silverstripe) site running on Ubuntu linux with apache and mysql.

Looking at the memory:

ps aux  | awk '{print $6/1024 " MB\t\t" $11}'  | sort -n

I get 150 entries for apache2 all with 40-90Mb. plus 475Mb for mysql and 331Mb for Java.

So it seems apache is starting too many processes and eating all the resources. Is that right?

If I restart apache, the site comes back up and works fine for an hour or so and then builds up lots of apache entries in ps and crashes.

how can i configure apache to not use up all the resources and avoid the website crashing? How can I find out what is causing the site resources to max out?

Update:

#/etc/apache2/mods-enabled# cat mpm_prefork.conf
<IfModule mpm_prefork_module>
    StartServers             5
    MinSpareServers       5
    MaxSpareServers      10
    MaxRequestWorkers     100
    MaxConnectionsPerChild   0
</IfModule>

Phpinfo

https://gist.github.com/asecondwill/bd9640cae780fa87a3af05be335aa3fb

Will
  • 131
  • 1
  • 6
  • When you check your logs, the reason for the increased load may simply be more real site visitors; but also look for abuse. Fairly typical abuse avenues are unpatched control panels, e-mail forms and open proxy configurations but abuse comes in many other forms as well. For example: https://serverfault.com/q/1130594/37681 – HBruijn Jun 05 '23 at 13:15
  • 1
    Note that simply adding the RSS memory reported by `ps` is over estimating the memory consumption because in each individual apache2 process the RSS includes and counts all shared libraries that are typically loaded only once in memory. - By default most LAMP distributions default to using the [prefork mpm](https://serverfault.com/questions/383526) and mod_php which does not offer the most performance when traffic increases. Switching to other MPM and PHP FPM offers improvement. – HBruijn Jun 05 '23 at 13:16
  • thanks @HBruijn - it wont be increased genunie traffic, the jump is too big. am I right looking at ps that the problem is too many apache threads / processes? so thats just too much traffic from some bots or something? and thats maxing out memory, triggering swap and maxing out cpu. – Will Jun 05 '23 at 13:28
  • ps over estimation: it adds up to 7g of memory, server has 4g. and not much else shows in the list, so number accuracy aside, its apache using the memory, and then swap using the cpu, right? – Will Jun 05 '23 at 13:42
  • Additional DB information request at 30 min uptime, please. OS, Version? # cores, any SSD or NVME devices on MySQL Host server? Post TEXT data on justpaste.it and share the links. From your SSH login root, Text results of: A) SELECT COUNT(*), sum(data_length), sum(index_length), sum(data_free) FROM information_schema.tables; B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) STATUS; not SHOW STATUS, just STATUS; G) SHOW ENGINE INNODB STATUS; for server workload tuning analysis to provide suggestions. – Wilson Hauck Jun 05 '23 at 15:26
  • Please edit your question to add your httpd configuration, especially MPM and PHP configuration. mod_php and prefork is very different from php-fpm and event. Please *do not* respond with these details in comments, the amount of information becomes unreadable without formatting. – John Mahowald Jun 05 '23 at 17:13
  • @JohnMahowald - ok thanks, mpm_prefork.conf added – Will Jun 05 '23 at 18:11
  • Please consider posting Additional DB information requested for workload analysis after 24 hours of uptime. – Wilson Hauck Jun 10 '23 at 23:29

1 Answers1

1

This goes way beyond the scope of an answer here. But some pointers might be helpful. First, don't try to measure memory usage using ps - reality is so much more complicated. Better to use 'free'. Limit the number of connection with max_clients.

On a web server, avoiding swap is really critical. If you have it switched on, turn it off now. Also limit the memory overcommit to capacity + 15% (and be prepared to revisit this).

Use mysql_tuner.pl to check your mysql config. Ensure that the memory is capped at a sensible level.

Since your main issue appears to be capacity, do make sure you are running php-fpm, not mod_php, and consider switching out Apache for nginx.

symcbean
  • 21,009
  • 1
  • 31
  • 52
  • Thanks for this. Switching to php-fpm and nginx are only going to make 10% max difference though right? We have gone from 50% server capacity to maxing out every 20mins. Feels like something is broken / hacked / misconfigured that needs fixing before just switching setups for optimisation – Will Jun 05 '23 at 13:26
  • Depends - if you are serving predominatly static content it could make 300% difference. If you are tunnelling Java traffic via Apache the difference will be much bigger too. – symcbean Jun 05 '23 at 13:32
  • its a php site (silverstripe). I suppose i could just vertically scale the server and see if that fixes it, and then optimise the site until we can turn the server back down. i think though something has gone wrong, rather than just needs a tweak – Will Jun 05 '23 at 13:41
  • *Also limit the memory overcommit to capacity + 15% (and be prepared to revisit this).* IMO there's not much point in allowing **any** memory overcommit unless you want to risk the OOM killer killing the web server process that's the entire reason the system exists in the first place, especially if it's using as much RAM as the OP states. Just disable memory overcommit entirely and make sure you have enough RAM to support what your services need. The nice thing about that is any later non-critical process that wants to hog RAM won't get it and it won't wake up the OOM killer. – Andrew Henle Jun 05 '23 at 14:28
  • IME 15% is conservative, especially on a PHP host. You can't disable overcommit - you can only limit it to 0%. – symcbean Jun 05 '23 at 14:31
  • turning off swap and setting overcommit - ill try these. where do I set them? – Will Jun 05 '23 at 15:38
  • `swapoff -a ; echo vm.overcommit_memory=2 >>/etc/sysctl.conf ; echo overcommit_ratio=15 >> /etc/systctl.conf ; sysctl -p` (memory changes will persist, to make swap change persist you need to edit /etc/fstab) – symcbean Jun 05 '23 at 15:45
  • Turning SWAP OFF is never a winner. It is much better to run slow for a few seconds than be faced with Out Of Memory error and termination of customers attempt to use your system. SWAP size should be 20% of RAM or 6GB minimum according to Red Hat - they have been in this long enough to know what is successful. – Wilson Hauck Jun 10 '23 at 00:50
  • Running in swap on a webserver is a recipe for failure and *entirely* unnecessary. – symcbean Jun 10 '23 at 18:35