5

I'm running a WHM/cPanel server on CentOS 6.6 with Apache 2.4 and PHP 5.5. Every week or so, CPU usage will shoot up to 100% across all six cores and stay there until Apache is restarted, at which point everything returns to normal. Interestingly, Apache's server-status page doesn't seem to know these processes exist:

Top:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25901 nobody    20   0 1973m  28m  276 R 74.8  0.4   3:39.30 httpd
24861 nobody    20   0 1973m  28m  280 R 74.1  0.4  12:05.31 httpd
25076 nobody    20   0 1973m  28m  276 R 65.8  0.4  10:09.38 httpd
24727 nobody    20   0 1973m  28m  280 R 64.5  0.4  14:37.09 httpd
25874 nobody    20   0 1973m  28m  276 R 64.5  0.4   3:57.69 httpd
24747 nobody    20   0 1973m  28m  276 R 64.1  0.4  15:06.89 httpd
25998 nobody    20   0 1973m  28m  276 R 63.8  0.4   2:40.92 httpd
25624 nobody    20   0 1973m  28m  276 R 61.8  0.4   5:28.76 httpd
25646 nobody    20   0 1973m  28m  276 R 58.8  0.4   5:07.88 httpd

Status page:

Server Version: Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4
Server MPM: event
Server Built: Mar 27 2015 11:20:11

Current Time: Tuesday, 09-Jun-2015 09:21:07 CDT
Restart Time: Tuesday, 02-Jun-2015 11:38:37 CDT
Parent Server Config. Generation: 12
Parent Server MPM Generation: 11
Server uptime: 6 days 21 hours 42 minutes 30 seconds
Server load: 8.17 7.35 10.46
Total accesses: 461541 - Total Traffic: 10.7 GB
CPU Usage: u111.81 s369.94 cu305989 cs438.15 - 51.4% CPU load
.774 requests/sec - 18.7 kB/second - 24.2 kB/request
7 requests currently being processed, 118 idle workers

PID     Connections     Threads   Async connections
        total accepting busy idle writing keep-alive closing
21715   1     yes       1    24   0       1          0
4766    0     yes       0    25   0       0          0
10222   0     yes       0    25   0       0          0
10278   6     yes       6    19   0       0          0
10194   0     yes       0    25   0       0          0
Sum     7               7    118  0       1          0
_____________________W__________________________________________
_____________W__W____W____W_W___W___.........................___
______________________

None of the requests reported by Apache's status page seem to be of any interest, which makes sense since none of the CPU-hogging PIDs are listed. Memory usage, disk I/O, and network traffic all remain relatively flat throughout, and the problem doesn't surface at a consistent time of day. There are dozens of small sites on this server, which would make searching through access logs by hand difficult.

What could be causing this? Am I just misunderstanding the way Apache reports data? Is there a better way to go about tracing the process responsible and seeing what it's actually doing?

Dan
  • 351
  • 1
  • 7
  • 18
  • what does full status look like; what does HTTP request look like which leads to this CPU hog? how long do these processes live is it seconds or 10+ seconds? – Hrvoje Špoljar Jun 16 '15 at 16:31
  • 1
    What is the full command line (press `c` in top or do a `ps -A -fg`) ? – Brian Jun 16 '15 at 17:53
  • 1
    Here's a lightly redacted full copy of `server-status`, along with the corresponding relevant `top` output with full command arguments: https://jsfiddle.net/xkxpgr2a/ – Dan Jun 16 '15 at 17:59
  • In addition using mod_status might help as detailed in the answer to a similar question: [Apache uses 100% CPU. Can “ps” command tell me what it is doing?](http://serverfault.com/questions/161405/apache-uses-100-cpu-can-ps-command-tell-me-what-it-is-doing) – Brian Jun 16 '15 at 17:59
  • @Brian: The server-status page *is* mod_status. A portion of its output is in my original question, and its full output is included in the fiddle above. – Dan Jun 16 '15 at 18:02
  • 1
    Config please. 1st. Turn off everything and try enabling one by one. 2nd. You won't be able to trace them cause they're run in context of web server. I would recommend changing apache-mod-php to nginx-fcgi-php for your security and scalability. – Vitaly Greck Jun 16 '15 at 22:25

4 Answers4

3

You can use debugging utility "strace" with CPU-hogging PIDs to see the cause of it. It may point you to the problem it has, strace -p <PID>

chetangb
  • 145
  • 6
  • strace looks promising, but the only output I get is the initial "Process attached" message, followed by silence. I also tried adding in the -c and -f flags, but those didn't change anything. – Dan Jun 18 '15 at 18:36
  • If `strace` is not printing anything out then the program is likely spinning in a loop without making system calls - ie it's not trying to read files or access the network. You can use `gdb` to attach to a running process and examine it for infinite loops, etc. Likely it won't be much help unless the running process was compiled with debugging information. – Chad Clark Jun 19 '15 at 19:12
  • 3
    strace doesn't usually have too much to say about cpu hogging because what it shows are system calls, and if that's what's slow, then the cpu would show up against the system, not the userland process. ltrace might be more useful, but it still won't show much if the work is in a tight php loop in your own code. A profiling tool like xtrace might be a better bet. – mc0e Jun 21 '15 at 21:42
0

There are 2 possible causes:

  1. Backups - cPanel backup process is a bit heavy, so first check if Apache load starts seconds/minutes after backup process start.

  2. Massive updates - more probable; with each day and week cPanel downloads huge number of different updates and checks, and during updates run many strange internal, heavy-weight programs, including license verifier, that is sometimes causing problems for some users.

Unfortunately cPanel's Apache is tied to heavy-weight cPanel CGI scripts, that do some parts of these updates and verifications. From my cPanel experience, I bet that exactly these CGI scripts are responsible for your Apache problems, because of deadlocks caused by interaction between them and cron jobs.

To check for both of these causes, disable cron jobs one by one by running as root:

crontab -e

Try to disable only one service at once and wait for a week, until you either see next high CPU usage, or find a problematic one.

Tomasz Klim
  • 458
  • 5
  • 10
  • We already keep a close eye on cPanel's backups and updates. They all run at night when traffic is low and finish within an hour. There are a couple smaller built-in scripts that run throughout the day, but they also all exit quickly. Both their CPU and disk I/O are always nominal. cPanel's license check doesn't happen through cron and AFAIK can't be disabled. – Dan Jun 16 '15 at 21:19
  • License check is done independently by many binaries, both standalone and CGI on Apache (I traced it so I'm sure). You can also try to disable KeepAlive in configuration and restart Apache, that's another common problem with cPanel. – Tomasz Klim Jun 16 '15 at 21:22
  • Disabling KeepAlive would most likely slow down sites on the server tremendously, so I'm hesitant to try it. The only way I can think that keepalives could cause symptoms like this is if a client were holding multiple connections open for hours at a time, and even that doesn't explain the high CPU usage or why none of the processes involved ever show up in Apache's server-status page. – Dan Jun 16 '15 at 21:31
  • Again, it's worth pointing out that these couple httpd processes are sitting at 100% CPU usage for **hours**, not seconds or even minutes. Life would make a lot more sense if they disappeared or calmed down eventually. – Dan Jun 16 '15 at 21:35
  • I mentioned KeepAlive, because there are reports from different users that KeepAlive, while being good itself, sometimes causes similar problems on cPanel Apache. – Tomasz Klim Jun 17 '15 at 06:02
0

Have you tried setting LogLevel debug and check {access,error}_log files for any hints?

Recently, I had to debug something in apache2 as well. What helped me was to stop apache2 service and start it manually using:

# strace -f -s 1024 -o /tmp/httpd.strace /usr/local/apache/bin/httpd -k start -DSSL -X

I took the full command line from your JSFiddle and just add the -X option to enable debug mode.

Once you hit the same situation, you can look at /tmp/httpd.strace for hints. It might be useful to use strace-graph /tmp/httpd.strace to see what sub-processes were invoked during the run of strace.

Srinidhi
  • 111
  • 3
0

use the PPIDs of the errant processes to track down the parent. I suspect you have two different apache daemons running. It seems likely that CPanel might do this in order to be able to do things as root, but I notice that the processes are user nobody. Maybe you have a lightweight apache that handles incopming requests and passes them to a second apache running heavier mod_php processes? Maybe there's something else going on, but your first task is to find what those apache processes are. Can you see two separate apache configurations?

lsof might be of use. You'd get info like which log files are open by a given apache process, and what port numbers it's listening on.

Assuming I'm right about it likely listening on a different port, it might be useful to set something up to capture traffic on that port so you can see what triggers the high cpu situation. Chances are it's no big deal to just leave tcpdump writing all that traffic to file, though you should be monitoring disk space just in case it's unexpectedly large.

It is interesting that you restarting apache does seem to work. MAybe I'm just wrong about there being two apaches, or perhaps there might be requests being forwarded between the apache instances.

sudo netstat -plnt might be interesting. It'll show you what processes and pids are associated with each listening port. If there's two apaches you'd find them in there. ps wwuaxf or pstree would also show you the apache processes grouped by parent process. You'd see the command line arguments

EDIT: extra after comment from OP.

Children of the init process are either separate apache instances running, or perhaps zombie processes that for some reason aren't being reaped. In that case the parent process has died, but the children haven't been able to be stopped and have been moved to having the init process as parent.

The high cpu might be something like repeated attempts to talk to the missing parent process, though in that case it'd presumably show up with strace. I'd look closely at what happens when apache is restarted.

Are some of the old processes left running? Do you have a good record of when the high CPU kicks in? (maybe use sar, munin, and kSar is good with sar, which otherwise has only text tables as an output). Can you correlate that with when apache restarts? (E.g. nightly log roll-over or manual actions). Maybe you can spot a connection with when something else happens on your system. If it's happening at the same time each time, that's very useful for tracking stuff down.

mc0e
  • 5,866
  • 18
  • 31
  • I'm pretty sure I've only got one copy of Apache that's supposed to be running, and I certainly don't maintain two separate Apache configs. `netstat` shows that none of the high-CPU processes are generating any network traffic, and `pstree` shows the troublesome Apache processes all being either direct children of `init(1)`, or children of another problem process that is (which is how Apache processes normally behave). However, `lsof` turns up what looks like an absolute treasure trove of information on what these processes are up to. It'll take some time to dig through, but it looks useful. – Dan Jun 23 '15 at 14:25