3

One of my servers, which runs in a german data center "hangs" every night, but i cant find out why. No errors are found in the /var/log/messages and /var/log/syslog.

The server responds to ping, but all services are down (ssh, apache, ...). After a reset everything runs normal.

A hardware test has been performed. It looks like being a software issue.

Martin Schlagnitweit
  • 141
  • 2
  • 6
  • Is it possible to log into local console when the server "hangs"? Examine `top` output. May be some process just takes almost all CPU time and all network services just get connection timeout. – HUB Jun 20 '11 at 16:27
  • Its possible to log into a local console, but a little bit complicated. This had been my next step. I now will log top output every minute like Eduardo suggested. – Martin Schlagnitweit Jun 20 '11 at 18:10
  • It seems to be a kernel panic. At the local console i can see this. But into which logfiles should this be written? – Martin Schlagnitweit Jun 21 '11 at 05:41
  • Is there "/var/log/kern.log" file? – HUB Jun 21 '11 at 09:22
  • In the kern.log file there are some extries like this: Jun 25 14:05:39 solunic kernel: [369632.475072] php-cgi[15194] general protection ip:6914c9 sp:7fffaf0f84d0 error:0 in php5-cgi[400000+6f9000] Today the server crahsed again at 16:00 CET, but the last error in kern.log was one hour before this. – Martin Schlagnitweit Jun 25 '11 at 15:27
  • top.log shows at the first line: 1378 mysql 20 0 354m 168m 4592 S 0.6 16.8 43:39.59 mysqld – Martin Schlagnitweit Jun 25 '11 at 15:31
  • the last entry of the sar command shows: 00:00:01 cpu %usr %nice %sys %irq %softirq %wait %idle _cpu_ ... 15:50:01 all 10 0 2 0 0 1 87 – Martin Schlagnitweit Jun 25 '11 at 15:32

3 Answers3

3

I'd leave some light profiling commands logging to files, so you can get an inside look on what went wrong after the fact. For example:

nohup top -b -d 60 >> top.log & # runs every 60 seconds
nohup vmstat 5 >> vmstat.log &
nohup iostat 5 >> iostat.log &

nohup is there so they aren't killed when you lose connection to the server. You can also use screen for that.

A more robust alternative to the last two commands would be to setup sar.

Eduardo Ivanec
  • 14,881
  • 1
  • 37
  • 43
  • I bet you meant to use ">>" instead of ">", right? – mfinni Jun 20 '11 at 17:08
  • It'd be better on some cases, but it's not necessary - each command keeps running and so opens/overwrites the output files only once at start. But I'll change it just in case, thanks! – Eduardo Ivanec Jun 20 '11 at 17:16
  • If those keep running after the "symptom" happens, then the output files will get over-written. That's why I'm suggesting the change. – mfinni Jun 20 '11 at 17:40
  • Oh wait - you mean the commands continue to run in the background, with the output redirected only once? Neat trick. – mfinni Jun 20 '11 at 17:46
  • Well, actually I meant the commands to be executed separately, and yes - they do keep running/cycling by themselves. But I'll add nohup and & because I wasn't that clear, thanks. – Eduardo Ivanec Jun 20 '11 at 18:58
1

When I have seen issues like this, it usually ends up being a problem with a cron job.

Check your syslog for cron jobs running at the same time of day that the server hangs. Also, check your root crontab (crontab -e) and jobs in /etc/cron.daily for anything that might be responsible.

Matt Beckman
  • 1,502
  • 18
  • 33
-2

Sounds like the random crashing could be caused by to faulty hardware. Have the hosting company see if there are any errors on POST or on the servers LCD. If it is a dell server you might want to install open manage which will tell you if any hardware is faulty. In my experience, a faulty memory dimm can cause random server reboots. Depending on what type of hardware you are running, it should be possible for your hosting to do a chassis swap on the server if the issue continues.

server_knuckle_dragger
  • 17
  • 2