Apache on my dedicated server constantly hangs due to connections being stuck in a state of "sending reply":
Apache status: connections stuck in "Sending Reply"
This happens at least several times a day - sometimes it won't happen for 2 or 3 hours, sometimes it'll happen constantly to the point where all connections will be occupied and stuck in sending reply within 5 minutes of restarting apache. This results in sites having constant downtime. The IP's in these requests are usually Amazon AWS or Cloudflare (I use Cloudflare for many sites, and Amazon AWS IP's may be from having the site using Ezoic nameservers) that access a variety of URL's on the site. The 'SS' column in the screenshot above shows how long the connections been stuck - in this example it's hundreds of seconds on each request, so something is very wrong.
These are simple Wordpress sites, usually no images, no unusual plugins, very simple and definitely nothing unusual that would cause a page to take a long time to load. However, they each have 5,000 to 100,000+ posts on each site and may be posting up to several thousand posts a day (using the plugin "WP all import pro") that are scheduled to publish at random times within 24 hours from being imported. There are 10+ sites like this on the server but only 3 or 4 constantly appear in apache status with stuck connections.
Across all sites on the server, monthly pageviews are barely over ~500,000 and should be easily handled by my server (from namecheap: XEON E3 1240 V3 8GB DDR3 1TB HDD SATA).
Mysql is often under high load on the server, and sometimes when looking at the processlist in phpmyadmin there will be stuck queries or sometimes NULL queries. Running the 'top' command also often shows very high CPU usage (my server has 4 CPU cores). Example screenshots:
Of concern is that the mysql error log in /var/lib/mysql is over 3GB in size now, too big for me to even read - renaming it and checking a new one generated shows many 'Got an error reading communication packets' errors. I don't know if that is the issue for stuck connections in apache and if so, what to do about it.
I have tried changing all possible apache settings such as Max Request Workers from the default 150 so more connections can be made (doesn't help as any new connections also get occupied and stuck), disabling all plugins or trying different themes on the wordpress sites (no effect), as well as changing some different mysql settings in my.cnf (per https://serverfault.com/a/989269/548654) with no improvement. I also tried changing all of the database tables for 1 of the problem sites to InnoDB from MyISAM and it still gets stuck connections. What else can I try to resolve this?
