1

I have a java application running on a CentOS 5.4 server.

Randomly, and seemingly without cause, these processes will die, not through the application exiting, or due to my killing it, but due to something that seems to kill without leaving a trace.

I log all output from the application, as well as sending stderr and stdout to a logfile, and none of these output logs contain anything that would indicate why these processes have died.

My first instinct was the kernel-level OOM killer, but the system is never low on memory, and usually has somewhere between 1GB and 3GB of memory free at any given point in time.

My question is: does anyone know what might be causing it, or does anyone have any ideas where I could start looking?

Thanks.

Martin
  • 55
  • 9

2 Answers2

1

Starting my java app with 

strace -tt -o strace.out <java command>

Showed that it was being sent a SIGHUP command which was killing it. Turns out that, although I was setting the process to run in the background (by appending & to the command), the command is still tied to whichever thread created it.

This meant that, for commands executed by apache, they would be killed whenever the worker thread that created them was recycled, and, for commands executed manually, they were killed whenever I logged out.

I remedied this by simply pre-pending nohup to the command, i.e.

nohup java -jar /path/to/my/java.jar arguments &
Martin
  • 55
  • 9
0

The processes may be getting some signal that causes them to exit, possibly due to bad hardware (e.g., bad memory can cause segfaults). If you see nothing the syslog/dmesg try enabling core dumps. You can then run gdb on them to get an idea what is happening.

Mark Wagner
  • 18,019
  • 2
  • 32
  • 47
  • I've built an identical setup on another of our servers and they're still being killed, so it's not faulty memory, and I'm still not getting any meaningful output. – Martin Feb 09 '11 at 09:09
  • It seems pressing return submits the comment. Interesting. I was going to ask - when you say "enabling core dumps" do you mean from Java or from the operating system? It seems Java has no warning when it's about to be killed. – Martin Feb 09 '11 at 09:10