I’m running several memory hungry Java server processes on Sun JVMs in a Solaris container. Recently I added more processes and may have exhausted the total available memory. Java processes started to crash randomly under load. In every single case it was a SIGSEGV in the JVM in thread GCTaskThread. At one point even top died with a SIGSEGV.
Does Solaris kill processes with SIGSEGV when memory gets short?
Instead of SIGSEGV I would have expected to get OutOfMemoryErrors at random locations. Also I always thought that segfaults in the JVM indicate bugs (unless JNI is involved). I need to know whether I have hit an additional JVM bug.
This is Solaris 10 and 32-bit JRE 1.6u20.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xfeb6297c, pid=26468, tid=6
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode solaris-sparc )
# Problematic frame:
# V [libjvm.so+0x76297c]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x0003bc00): GCTaskThread [stack: 0x8b300000,0x8b380000] [id=6]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x8b828bfb
Registers:
O0=0xf824d1d0 O1=0xdcdc43e8 O2=0xf824d1d3 O3=0x00000001
O4=0xfee2d800 O5=0x00000003 O6=0x8b37f718 O7=0xfe546d18
G1=0xfe55a854 G2=0x90d7f7b4 G3=0xfffffff4 G4=0x00000000
G5=0xfee2dd5f G6=0x00000000 G7=0xfeed2200 Y=0x00000000
PC=0xfeb6297c nPC=0xfeb62980
Top of Stack: (sp=0x8b37f718)
0x8b37f718: 8bc00000 00033158 fee3f000 fbc00000
0x8b37f728: 70000000 1c000000 8b3a2000 00033200
0x8b37f738: fee3b800 00486bfb 00000011 00000003
0x8b37f748: f824d1d3 dbc00000 8b37f778 fe55a854
0x8b37f758: ed204000 00001000 00000000 00000032
0x8b37f768: 8737f7f0 0003c3a8 8b37f7ec 0191c568
0x8b37f778: 0003c3a8 00000010 00000003 00000003
0x8b37f788: 00000009 8b37f84c 00000004 8b37f84c
Instructions: (pc=0xfeb6297c)
0xfeb6296c: ee 06 21 b8 b3 36 60 09 b4 10 20 11 ec 05 e0 4c
0xfeb6297c: f4 2d 80 19 81 c7 e0 08 81 e8 20 00 9d e3 bf a0
Stack: [0x8b300000,0x8b380000], sp=0x8b37f718, free space=1fdfeb6297ck
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x76297c]
V [libjvm.so+0x15a85c]
V [libjvm.so+0x486acc]
V [libjvm.so+0x169314]
V [libjvm.so+0x16bda0]
V [libjvm.so+0x24c2a4]
V [libjvm.so+0x72a1a0]
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x027bf800 JavaThread "pool-3-thread-31" [_thread_blocked, id=1171, stack(0x90d00000,0x90d80000)]
0x028df000 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=1169, stack(0x91400000,0x91480000)]
0x028dec00 JavaThread "AWT-Shutdown" [_thread_blocked, id=1168, stack(0x91700000,0x91780000)]
0x00dd4800 JavaThread "Product-producer-thread-7" [_thread_blocked, id=1165, stack(0x90b00000,0x90b80000)]
0x03a18400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" [_thread_in_native, id=580, stack(0x85100000,0x85180000)]
0x00f83c00 JavaThread "BaseComTimerThread-9" daemon [_thread_blocked, id=287, stack(0x88d80000,0x88e00000)]
0x014d5000 JavaThread "PACCommand_7" [_thread_blocked, id=105, stack(0x86500000,0x86580000)]
0x0109b800 JavaThread "PACCommand_6" [_thread_blocked, id=104, stack(0x86600000,0x86680000)]
0x0109b000 JavaThread "PACCommand_5" [_thread_blocked, id=103, stack(0x86700000,0x86780000)]
0x00d75800 JavaThread "PACCommand_4" [_thread_blocked, id=102, stack(0x86800000,0x86880000)]
0x0106ac00 JavaThread "PACCommand_3" [_thread_blocked, id=101, stack(0x86900000,0x86980000)]
0x01529000 JavaThread "PACCommand_2" [_thread_blocked, id=100, stack(0x86a00000,0x86a80000)]
0x0131e800 JavaThread "PACCommand_1" [_thread_blocked, id=99, stack(0x86b00000,0x86b80000)]
0x01163400 JavaThread "BaseComSingleThread-1" daemon [_thread_blocked, id=91, stack(0x86d00000,0x86d80000)]
0x002f6800 JavaThread "FailoverExecutorThread-0" daemon [_thread_blocked, id=90, stack(0x87500000,0x87580000)]
0x00dd6800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=89, stack(0x86f00000,0x86f80000)]
0x00b8fc00 JavaThread "NonGUI_5" [_thread_blocked, id=87, stack(0x86e00000,0x86e80000)]
0x0091d000 JavaThread "NonGUI_4" [_thread_blocked, id=86, stack(0x87000000,0x87080000)]
0x0191e800 JavaThread "NonGUI_3" [_thread_blocked, id=82, stack(0x87100000,0x87180000)]
0x01fa0400 JavaThread "NonGUI_2" [_thread_blocked, id=81, stack(0x87200000,0x87280000)]
0x018f6800 JavaThread "NonGUI_1" [_thread_blocked, id=80, stack(0x87300000,0x87380000)]
0x00ef0400 JavaThread "Timer legends" [_thread_blocked, id=79, stack(0x87400000,0x87480000)]
0x01179400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=75, stack(0x87600000,0x87680000)]
0x009bd800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=73, stack(0x87700000,0x87780000)]
0x00931800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=71, stack(0x88980000,0x88a00000)]
0x00a6e400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=69, stack(0x88a80000,0x88b00000)]
0x0113b400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=67, stack(0x88b80000,0x88c00000)]
0x00c72000 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=65, stack(0x87900000,0x87980000)]
0x001bbc00 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=63, stack(0x87a00000,0x87a80000)]
0x00f45c00 JavaThread "Access Layer MDT Update Thread #0" daemon [_thread_blocked, id=60, stack(0x89900000,0x89980000)]
0x00a70c00 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=59, stack(0x87800000,0x87880000)]
0x00a68800 JavaThread "AWT-XAWT" daemon [_thread_in_native, id=53, stack(0x87b00000,0x87b80000)]
0x008ecc00 JavaThread "ProductTimer" [_thread_blocked, id=52, stack(0x87d00000,0x87d80000)]
0x00030400 JavaThread "DestroyJavaVM" [_thread_blocked, id=2, stack(0xfe300000,0xfe380000)]
0x00e34c00 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" [_thread_in_native, id=48, stack(0x87e00000,0x87e80000)]
0x00e32800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=47, stack(0x87f00000,0x87f80000)]
0x00f68400 JavaThread "Proxy.cleaner" daemon [_thread_blocked, id=45, stack(0x88000000,0x88080000)]
0x00f67400 JavaThread "Thread-10" [_thread_blocked, id=44, stack(0x88100000,0x88180000)]
0x00f66400 JavaThread "Timer-1" [_thread_blocked, id=43, stack(0x88200000,0x88280000)]
0x00f64c00 JavaThread "CORBA cleaner" daemon [_thread_blocked, id=42, stack(0x88300000,0x88380000)]
0x00f64400 JavaThread "Timer-0" [_thread_blocked, id=41, stack(0x88400000,0x88480000)]
0x0117e800 JavaThread "BaseComExecutorThread-4" [_thread_blocked, id=40, stack(0x88500000,0x88580000)]
0x01fbd400 JavaThread "BaseComExecutorThread-3" [_thread_blocked, id=39, stack(0x88600000,0x88680000)]
0x011ad400 JavaThread "BaseComExecutorThread-2" [_thread_blocked, id=38, stack(0x88700000,0x88780000)]
0x011acc00 JavaThread "Product.scheduler.thread" [_thread_blocked, id=37, stack(0x88800000,0x88880000)]
0x00a9f800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=35, stack(0x88e80000,0x88f00000)]
0x00e95000 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" [_thread_in_native, id=34, stack(0x89400000,0x89480000)]
0x00c14400 JavaThread "BaseComExecutorThread-1" [_thread_blocked, id=32, stack(0x89500000,0x89580000)]
0x00e92000 JavaThread "ConfigCacheCleaner4" daemon [_thread_blocked, id=31, stack(0x89600000,0x89680000)]
0x00f4e400 JavaThread "ConfigCacheCleaner3" daemon [_thread_blocked, id=30, stack(0x89700000,0x89780000)]
0x00cb6000 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=29, stack(0x89800000,0x89880000)]
0x00ba0800 JavaThread "BaseComTimerThread-1" daemon [_thread_blocked, id=25, stack(0x89a00000,0x89a80000)]
0x00133000 JavaThread "TaskRouterThread-1" daemon [_thread_blocked, id=24, stack(0x89b00000,0x89b80000)]
0x00f3c000 JavaThread "NINJO_CACHE_1333544439197_ScratchConfigCache1.3_2.data" daemon [_thread_blocked, id=23, stack(0x89c00000,0x89c80000)]
0x00c00400 JavaThread "NINJO_CACHE_1333544439197_ScratchConfigCache1.3_1.data" daemon [_thread_blocked, id=22, stack(0x89d00000,0x89d80000)]
0x00ed6c00 JavaThread "NINJO_CACHE_1333544439197_ScratchConfigCache1.3.data" daemon [_thread_blocked, id=21, stack(0x89e00000,0x89e80000)]
0x00ba3c00 JavaThread "com.product.fwk.common.cache.impl.EHCacheManager@b02928" daemon [_thread_blocked, id=20, stack(0x89f00000,0x89f80000)]
0x00aa3400 JavaThread "com.product.fwk.common.cache.impl.EHCacheManager@b02928" daemon [_thread_blocked, id=19, stack(0x8a000000,0x8a080000)]
0x00ab5400 JavaThread "ConfigCacheCleaner2" daemon [_thread_blocked, id=18, stack(0x8a100000,0x8a180000)]
0x00c0d800 JavaThread "ConfigCacheCleaner1" daemon [_thread_blocked, id=17, stack(0x8a200000,0x8a280000)]
0x00e40000 JavaThread "ORB-Thread" [_thread_blocked, id=16, stack(0x8a300000,0x8a380000)]
0x00e3e800 JavaThread "ORBacus:Server:StarterThread" [_thread_in_native, id=15, stack(0x8a400000,0x8a480000)]
0x00127c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=13, stack(0x8aa80000,0x8ab00000)]
0x00126000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=12, stack(0x8ab80000,0x8ac00000)]
0x00123800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=11, stack(0x8ac80000,0x8ad00000)]
0x00122000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=10, stack(0x8ad80000,0x8ae00000)]
0x0010d000 JavaThread "Finalizer" daemon [_thread_blocked, id=9, stack(0x8ae80000,0x8af00000)]
0x0010b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=8, stack(0x8af80000,0x8b000000)]
Other Threads:
0x00109400 VMThread [stack: 0x8b080000,0x8b100000] [id=7]
0x0012ac00 WatcherThread [stack: 0x8a980000,0x8aa00000] [id=14]
=>0x0003bc00 (exited) GCTaskThread [stack: 0x8b300000,0x8b380000] [id=6]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x0002f578] Threads_lock - owner thread: 0x00109400
[0x0002f988] Heap_lock - owner thread: 0x00dd4800
Heap
PSYoungGen total 457280K, used 83928K [0xdbc00000, 0xfbc00000, 0xfbc00000)
eden space 391808K, 21% used [0xdbc00000,0xe0df6178,0xf3aa0000)
from space 65472K, 0% used [0xf3aa0000,0xf3aa0000,0xf7a90000)
to space 61632K, 4% used [0xf7fd0000,0xf8254130,0xfbc00000)
PSOldGen total 1048576K, used 331538K [0x9bc00000, 0xdbc00000, 0xdbc00000)
object space 1048576K, 31% used [0x9bc00000,0xaffc4b68,0xdbc00000)
PSPermGen total 49152K, used 45797K [0x8bc00000, 0x8ec00000, 0x9bc00000)
object space 49152K, 93% used [0x8bc00000,0x8e8b9690,0x8ec00000)
Dynamic libraries:
0x00010000 /opt/product/release_1.6/server/jre/bin/java
0xff3a0000 /lib/libthread.so.1
0xff370000 /opt/product/release_1.6/server/jre/bin/../lib/sparc/jli/libjli.so
0xff350000 /lib/libdl.so.1
0xff200000 /lib/libc.so.1
0xff390000 /platform/SUNW,SPARC-Enterprise/lib/libc_psr.so.1
0xfe400000 /opt/product/release_1.6/server/jre/lib/sparc/server/libjvm.so
0xff1d0000 /lib/libsocket.so.1
0xff1f0000 /usr/lib/libsched.so.1
0xff1b0000 /lib/libm.so.1
0xff180000 /usr/lib/libCrun.so.1
0xff160000 /lib/libdoor.so.1
0xff080000 /lib/libnsl.so.1
0xfef80000 /lib/libm.so.2
0xff050000 /lib/libscf.so.1
0xff140000 /lib/libuutil.so.1
0xff030000 /lib/libgen.so.1
0xfef50000 /lib/libmd.so.1
0xfef30000 /lib/libmp.so.2
0xfeea0000 /opt/product/release_1.6/server/jre/lib/sparc/libverify.so
0xfee60000 /opt/product/release_1.6/server/jre/lib/sparc/libjava.so
0xfe3e0000 /opt/product/release_1.6/server/jre/lib/sparc/native_threads/libhpi.so
0xfe2e0000 /lib/nss_files.so.1
0xfe2b0000 /usr/lib/nss_vas4.so.1
0xfe280000 /opt/quest/lib/libvtsmartcache.so.1
0xfe180000 /opt/quest/lib/libvtutil.so.1
0xfe230000 /opt/quest/lib/libvtcacheipc.so.1
0xfe120000 /lib/libresolv.so.2
0xfe100000 /opt/product/release_1.6/server/jre/lib/sparc/libzip.so
0xfe0b0000 /usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3
0x8ae50000 /opt/product/release_1.6/server/jre/lib/sparc/libnet.so
0x89200000 /opt/product/release_1.6/server/jre/lib/sparc/libawt.so
0x8ad20000 /opt/product/release_1.6/server/jre/lib/sparc/xawt/libmawt.so
0x8ae10000 /usr/openwin//lib/libXext.so.0
0x89080000 /usr/openwin//lib/libX11.so.4
0x8ac50000 /usr/openwin//lib/libXtst.so.1
0x8ac20000 /usr/openwin//lib/libXi.so.5
0x88f80000 /opt/product/release_1.6/server/jre/lib/sparc/libfontmanager.so
0x8b110000 /opt/product/release_1.6/server/jre/lib/sparc/libnio.so
0x8ab60000 /lib/librt.so.1
0x8ab40000 /lib/libaio.so.1
0x8af10000 /usr/lib/libsendfile.so.1
0x887d0000 /opt/product/release_1.6/server/jre/lib/sparc/libmanagement.so
0x887b0000 /usr/openwin/lib/locale/common/xlibi18n.so.2
0x886d0000 /usr/X11/lib/libXcursor.so.1
0x886a0000 /usr/openwin/sfw/lib/libXrender.so.1
0x8a850000 /opt/product/release_1.6/server/jre/lib/sparc/libdcpr.so
VM Arguments:
jvm_args: -Xmx1536M -Xms1536M -XX:MaxPermSize=256m -XX:-ReduceInitialCardMarks -Djava.ext.dirs=/opt/product/release_1.6/server/lib/ext -Djava.endorsed.dirs=/opt/product/release_1.6/server/lib/boot -Dorg.omg.CORBA.ORBClass=com.ooc.CORBA.ORB -Dorg.omg.CORBA.ORBSingletonClass=com.ooc.CORBA.ORBSingleton -Dcorba.port=23801 -Dserver.name=ProductServer -Dserver.basedir=/data/product -Dproduct.log.path=../../log -Dproduct.i18n.locale.language=en
java_command: /opt/product/release_1.6/server/lib/product/ProductMain.jar /opt/product/release_1.6/server/config.properties
Launcher Type: SUN_STANDARD
Environment Variables:
CLASSPATH=/opt/local/opkg/share/oracle/product/10.2.0/jdbc/lib/classes12.zip
PATH=/usr/bin:/usr/sbin:/usr/local/bin:/opt/local/opkg/bin:/opt/local/opkg/sbin:/home/ext/user/bin:/opt/local/opkg/share/oracle/product/10.2.0/bin:/usr/bin:/usr/sbin:/usr/local/bin:/opt/local/opkg/bin:/opt/local/opkg/sbin:/home/users/user/bin:/usr/bin:/home/users/user/bin
LD_LIBRARY_PATH=/opt/product/release_1.6/server/jre/lib/sparc/server:/opt/product/release_1.6/server/jre/lib/sparc:/opt/product/release_1.6/server/jre/../lib/sparc:/opt/local/opkg/share/oracle/product/10.2.0/lib32:/opt/local/opkg/share/oracle/product/10.2.0/lib:/opt/local/opkg/share/oracle/product/10.2.0/jdbc/lib32
SHELL=/bin/bash
DISPLAY=localhost:63.0
Signal Handlers:
SIGSEGV: [libjvm.so+0x84dd5c], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0x84dd5c], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGXFSZ: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x72c934], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x72c934], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIG39: [libjvm.so+0x730360], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIG40: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
--------------- S Y S T E M ---------------
OS: Solaris 10 10/09 s10s_u8wos_08a SPARC
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 16 September 2009
uname:SunOS 5.10 Generic_142909-17 sun4u (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 8192, AS infinity
load average:7.79 3.58 3.31
CPU:total 4 has_v8, has_v9, popc, has_vis1, has_vis2, is_ultra3
Memory: 8k page, physical 15728640k(1399920k free)
vm_info: Java HotSpot(TM) Server VM (16.3-b01) for solaris-sparc JRE (1.6.0_20-b02), built on Apr 12 2010 14:27:21 by "" with Workshop 5.8
time: Wed Apr 4 13:50:18 2012
elapsed time: 2982 seconds
