5

One of our dev servers keeps crashing every now and then and the reports look very similar. We're thinking it is due to lack of memory, but we want to verify this. Could you guys assist in this process? Below you'll find the relevant information from the hs_err file.

Thanks! Yon

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0x00002b84b6dee37c, pid=4196, tid=1081399616
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_22-b03 mixed mode)
# Problematic frame:
# V  [libjvm.so+0x5b437c]
#

---------------  T H R E A D  ---------------

Current thread (0x000000005db44970):  GCTaskThread [id=4200]

siginfo:si_signo=11, si_errno=0, si_code=128, si_addr=0x0000000000000000


Heap
 PSYoungGen      total 291968K, used 291760K [0x00002aaada600000, 0x00002aaaec400000, 0x00002aaaec400000)
  eden space 291136K, 100% used [0x00002aaada600000,0x00002aaaec250000,0x00002aaaec250000)
  from space 832K, 75% used [0x00002aaaec250000,0x00002aaaec2ec288,0x00002aaaec320000)
  to   space 896K, 21% used [0x00002aaaec320000,0x00002aaaec350000,0x00002aaaec400000)
 PSOldGen        total 583680K, used 385757K [0x00002aaab6c00000, 0x00002aaada600000, 0x00002aaada600000)
  object space 583680K, 66% used [0x00002aaab6c00000,0x00002aaace4b7438,0x00002aaada600000)
 PSPermGen       total 116736K, used 116682K [0x00002aaaaac00000, 0x00002aaab1e00000, 0x00002aaab6c00000)
  object space 116736K, 99% used [0x00002aaaaac00000,0x00002aaab1df2b78,0x00002aaab1e00000)


---------------  S Y S T E M  ---------------

OS:CentOS release 5.3 (Final)

uname:Linux 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64
libc:glibc 2.5 NPTL 2.5 
rlimit: STACK 10240k, CORE 0k, NPROC 16384, NOFILE 99999, AS infinity
load average:22.73 19.62 19.08

CPU:total 4 em64t

Memory: 4k page, physical 2059636k(196532k free), swap 128512k(120972k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (1.5.0_22-b03) for linux-amd64, built on Oct  9 2009 01:32:14 by java_re with gcc 3.2.2 (SuSE Linux)

time: Fri Aug  5 03:57:27 2011
elapsed time: 27420 seconds
skaffman
  • 398,947
  • 96
  • 818
  • 769
Yon
  • 1,023
  • 4
  • 13
  • 23
  • Upgrading to a modern 1.6.x JVM is the obvious (and sensible) suggestion – skaffman Aug 07 '11 at 09:30
  • Not possible, this is a product that's deployed with numerous customers. Any other ideas? – Yon Aug 08 '11 at 13:17
  • 1
    Your JVM is crashing. The bug is in the JVM. You're using an old JVM. The bug is likely fixed in a newer JVM. It may not be the answer you wanted, but the conclusion is unavoidable. – skaffman Aug 08 '11 at 13:21
  • It didn't happen in the past, it's fairly new, with the same server, JVM, etc. So we probably triggered this behavior somehow. Is there no known list of issues of this type an how to avoid them? – Yon Aug 08 '11 at 16:50

2 Answers2

3

As a work around, you can increase size of perm gen by "-XX:PermSize=256m -XX:MaxPermSize=256m" . It doesn't fix the problem, but it will delay the crash.

Or you can try other gc policy like concurrent gc .

Kuai Wei
  • 86
  • 4
1

Lack of memory should not cause JVM crashes. If it does, that is a JVM bug, and the only real fix for a JVM bug is to upgrade.

The only possibilities I can think of where this is "your fault" are:

  • your code or some third-party library is using native code libraries for something, and that code is buggy,

  • your JVM installation has been subtly corrupted, or

  • you've got an intermittent hardware fault on that machine.

If you suspect that the problem is lack of memory, then running the application with GC logging enabled may confirm this. Or you could just tweak the heap sizes and other settings and hope that they fix it.

At some point you are going to have to tell your customers that you can no longer provide support for installations on old (end-of-life'd) JVMs. If this is a JVM bug (as we suspect) then there is little chance of getting a fix for it ... unless you / your customers are willing to sign a BIG cheque to Oracle for support.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • We will be moving to Java 6 in Feb 2012. The report above shows that eden space is at 100%, PermGen is at 99% and the fault is at the GCThread. Are we really the ones ones to ever encounter this? – Yon Aug 14 '11 at 06:11
  • @Yon - *"Are we really the ones ones to ever encounter this?"* Probably not. But the best fix is to upgrade. – Stephen C Aug 14 '11 at 07:40