fork: Cannot allocate memory

Question

I'm having some problem with a couple of servers having quite much amount of connections.

The servers just hangs for new network connections (although the functionality of load balancing in haproxy seems to work correct). I can not connect to ssh, and server cannot connect to other sockets.

I stayed connected with one console on screen to server. Then the error occurred I was not able to execute any command via screen.

# ls
-bash: fork: Cannot allocate memory 

I could not execute "free -m" to show number of memory, but I got monitoring implemented which indicated, that the last value send was (RAM free, Buffers free, Swap free) (3597MB, 5793, 5783).

Server has many TIME_WAIT connection in netstat despite I changes tcp_fin_timeout to 15 (9155 TIME_WAIT, 55 ESTABLISHED). I was only able to run ulimits -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 88064
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 88064
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited 

PS. It's not a VPS. It's DELL.

Answer 1

If you have an idea which processes cause all this (i.e. Apache) you could try to recover from this situation with the command exec killall -9 httpd. This replaces the current shell with the killall program, which hopefully eliminates the offending processes, allowing you to further analyse the problem. Beware though, this is a one-shot attempt, after that the shell in your screen session will be gone so if this fails, you will need a hard reboot.