2

I'm having some problem with a couple of servers having quite much amount of connections.

The servers just hangs for new network connections (although the functionality of load balancing in haproxy seems to work correct). I can not connect to ssh, and server cannot connect to other sockets.

I stayed connected with one console on screen to server. Then the error occurred I was not able to execute any command via screen.

# ls
-bash: fork: Cannot allocate memory 

I could not execute "free -m" to show number of memory, but I got monitoring implemented which indicated, that the last value send was (RAM free, Buffers free, Swap free) (3597MB, 5793, 5783).

Server has many TIME_WAIT connection in netstat despite I changes tcp_fin_timeout to 15 (9155 TIME_WAIT, 55 ESTABLISHED). I was only able to run ulimits -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 88064
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 88064
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited 

PS. It's not a VPS. It's DELL.

krogon
  • 21
  • 1
  • 2

1 Answers1

3

If you have an idea which processes cause all this (i.e. Apache) you could try to recover from this situation with the command exec killall -9 httpd. This replaces the current shell with the killall program, which hopefully eliminates the offending processes, allowing you to further analyse the problem. Beware though, this is a one-shot attempt, after that the shell in your screen session will be gone so if this fails, you will need a hard reboot.

Sven
  • 98,649
  • 14
  • 180
  • 226
  • 1
    old question and old answer, but much appreciated. Saved my server from a hard reset – Rob Aug 02 '12 at 22:58