I'm running a set of rails apps on a server under Apache/Passenger. One of the apps keeps getting into a state where the load on the machine gets very high and the machine becomes unresponsive with top showing that most of the CPU time is spent on IO wait but none of the processes are using much time. iowait and ifwait don't show anything out of the ordinary. passenger-status is the most interesting, it shows every single applications process for what I suspect is the problem app (app x in the output below) is occupied with 1 session. I can kill all of these processes and passenger replaces them with no effect on server load, only when I restart apache or the server does load return to normal. Here's the output:
----------- General information -----------
max = 20
count = 20
active = 16
inactive = 4
Waiting on global queue: 22
----------- Application groups -----------
/home/A:
App root: /home/A
* PID: 5379 Sessions: 0 Processed: 6 Uptime: 3m 42s
/home/X:
App root: /home/X
* PID: 5604 Sessions: 1 Processed: 2 Uptime: 1m 8s
* PID: 5560 Sessions: 1 Processed: 3 Uptime: 1m 10s
* PID: 5416 Sessions: 1 Processed: 5 Uptime: 3m 30s
* PID: 5387 Sessions: 1 Processed: 19 Uptime: 3m 30s
* PID: 5571 Sessions: 1 Processed: 2 Uptime: 1m 9s
* PID: 5408 Sessions: 1 Processed: 6 Uptime: 3m 30s
* PID: 5424 Sessions: 1 Processed: 3 Uptime: 3m 30s
* PID: 5306 Sessions: 1 Processed: 18 Uptime: 3m 30s
* PID: 5553 Sessions: 1 Processed: 1 Uptime: 1m 10s
* PID: 5612 Sessions: 1 Processed: 1 Uptime: 1m 8s
* PID: 5394 Sessions: 1 Processed: 4 Uptime: 3m 30s
* PID: 5401 Sessions: 1 Processed: 14 Uptime: 3m 30s
* PID: 5620 Sessions: 1 Processed: 1 Uptime: 1m 8s
* PID: 5298 Sessions: 1 Processed: 7 Uptime: 4m 33s
* PID: 5580 Sessions: 1 Processed: 2 Uptime: 1m 9s
* PID: 5594 Sessions: 1 Processed: 1 Uptime: 1m 8s
/home/B:
App root: /home/B
* PID: 5362 Sessions: 0 Processed: 3 Uptime: 4m 2s
/home/C:
App root: /home/C
* PID: 5332 Sessions: 0 Processed: 5 Uptime: 4m 18s
* PID: 5369 Sessions: 0 Processed: 10 Uptime: 3m 30s
