1

Hey I'm having a weird issue that I'm having some difficulty diagnosing. I have a VM that went down due to high CPU usage today. The websites on this VM are very low traffic, so I knew something unusual was at play. I ran several basic diagnostic commands and determined that a User Process(es) was using excessive CPU. So then I ran top, and found something unusual. While my CPU usage is being reported high for User Processes, the top processes are all using surprisingly little CPU. I'm having difficulty pinpointing the issue without a clear focus.

Here are the first several lines of the top output:

Tasks: 117 total,   2 running, 115 sleeping,   0 stopped,   0 zombie
Cpu0  : 73.1%us, 10.0%sy,  0.0%ni, 16.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 84.7%us, 10.3%sy,  0.0%ni,  4.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Mem:   2054472k total,   622260k used,  1432212k free,    59960k buffers
Swap:        0k total,        0k used,        0k free,   237724k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                      
 5926 bay  20   0  161m  18m 5384 R  2.7  0.9   0:00.08 php                                                                                                           
 1482 mysql     20   0  932m  42m 5996 S  2.3  2.1   0:04.79 mysqld                                                                                                        
 3197 nobody    20   0 71824 3788 1628 S  0.3  0.2   0:00.09 httpd                                                                                                         
    1 root      20   0 19232 1488 1220 S  0.0  0.1   0:00.59 init                                                                                                          
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                                      
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                                                   
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/0                                                                                                   
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                                                   
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                                                                                                    
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1                                                                                                   
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1          

As you can see the processes listed come nowhere near the 73/84% CPU usage. Any ideas as to how I can see these missing processes or how I can better pinpoint my issue? Thanks a lot guys!

  • 3
    Kinda feels like you might be pwned, and the dodgy processes are being hidden from `/proc` by a kernel module. See [this canonical question](http://serverfault.com/q/218005/1375) for more info on investigating a compromise. – womble Aug 26 '15 at 03:45

0 Answers0