Segmentation fault in GC Thread

Question

We've deployed our Dropwizard service on aws-ec2 with centos7.

We are intermittently facing with segmentation fault on our GC thread.

Currently we are taking thread-pumps every minute to analyze.

Current GC Algo - CMS

What we tried so far:

Changing OS from Amazon linux to centos7
Tried different GC algos - parallel GC, CMS, G1-GC

Please find the stack trace below.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f4be96146bb, pid=1016, tid=139963858630400
#
# JRE version: Java(TM) SE Runtime Environment (8.0_72-b15) (build 1.8.0_72-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x6516bb]  CollectedHeap::common_mem_allocate_init(KlassHandle, unsigned long, Thread*)+0x9b
#
# Core dump written. Default location: //core or core.1016
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------
Current thread (0x00007f4be40d7800):  GCTaskThread [stack: 0x00007f4be0034000,0x00007f4be0134000] [id=3424]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000010000000b

Registers:
RAX=0x00007f4be9f68c19, RBX=0x00007f4be40d7800, RCX=0x0000000000000000, RDX=0x00007f4be40d7800
RSP=0x00007f4be0131920, RBP=0x00007f4be0131950, RSI=0x0000000000000003, RDI=0x00000007c00016d0
R8 =0x0000000000000000, R9 =0x0000000000000000, R10=0x202e64657265746e, R11=0x0000000000000001
R12=0x00000000ffffffff, R13=0x0000000000000003, R14=0x00000007c00016d0, R15=0x0000000010000031
RIP=0x00007f4be96146bb, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e
...

JavaThread 0x00007f4adc006800 (nid = 13003) was being processed
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  ~RuntimeStub::_new_array_nozero_Java
J 3340 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (86 bytes) @ 0x00007f4bb99ea98c [0x00007f4bb99ea2c0+0x6cc]
J 4234 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (97 bytes) @ 0x00007f4bb9cfc970 [0x00007f4bb9cfc8e0+0x90]
J 5849 C2 java.net.URLClassLoader$2.run()Ljava/lang/Object; (5 bytes) @ 0x00007f4bb9360d7c [0x00007f4bb9360c20+0x15c]
v  ~StubRoutines::call_stub
J 5043  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007f4bb9e2a563 [0x00007f4bb9e2a500+0x63]
J 5846 C2 java.lang.ClassLoader.getResource(Ljava/lang/String;)Ljava/net/URL; (87 bytes) @ 0x00007f4bb98f51d4 [0x00007f4bb98f4ee0+0x2f4]
J 26710 C2 javax.xml.soap.FactoryFinder.find(Ljava/lang/String;Ljava/lang/String;Z)Ljava/lang/Object; (317 bytes) @ 0x00007f4bbacd220c [0x00007f4bbacd18c0+0x94c]
J 27488 C2 javax.xml.soap.SAAJMetaFactory.getInstance()Ljavax/xml/soap/SAAJMetaFactory; (44 bytes) @ 0x00007f4bbab9d3ac [0x00007f4bbab9d380+0x2c]
J 21785 C1 javax.xml.soap.MessageFactory.newInstance(Ljava/lang/String;)Ljavax/xml/soap/MessageFactory; (8 bytes) @ 0x00007f4bbbe04b1c [0x00007f4bbbe04ac0+0x5c]
j  com.sun.xml.internal.messaging.saaj.client.p2p.HttpSOAPConnection.<init>()V+18
j  com.sun.xml.internal.messaging.saaj.client.p2p.HttpSOAPConnectionFactory.createConnection()Ljavax/xml/soap/SOAPConnection;+4
J 24605 C1 com.tp.pg.util.shopping.SoapWSCall.getToken(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;I)Ljava/util/Map; (160 bytes) @ 0x00007f4bbc4be4d4 [0x00007f4bbc4be380+0x154]
J 24677 C1 com.tp.pg.util.shopping.SoapWSCall.getTransactionToken(Ljava/lang/String;Ljava/lang/String;I)Ljava/util/Map; (452 bytes) @ 0x00007f4bbdbd060c [0x00007f4bbdbced80+0x188c]
J 24279 C1 com.tp.pg.util.TransactionRequestBean.getTransactionToken()Ljava/lang/String; (259 bytes) @ 0x00007f4bbc40132c [0x00007f4bbc4005e0+0xd4c]
J 27355 C1 com.games24x7.paymentservice.util.TechprocessReconciliationTask.getTechProcessToken(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;I)Ljava/lang/String; (321 bytes) @ 0x00007f4bbdf815bc [0x00007f4bbdf7f740+0x1e7c]
J 27354 C1 com.games24x7.paymentservice.util.TechprocessReconciliationTask.getReconcilationRequest(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;I)Lorg/json/JSONObject; (176 bytes) @ 0x00007f4bbb8014f4 [0x00007f4bbb801420+0xd4]
j  com.games24x7.paymentservice.util.TechprocessReconciliationTask.run()V+166
J 16559 C2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (14 bytes) @ 0x00007f4bbaa327ec [0x00007f4bbaa327a0+0x4c]
J 19385 C2 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007f4bbcdef074 [0x00007f4bbcdeefc0+0xb4]
J 21343% C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f4bbc1fbc0c [0x00007f4bbc1fb980+0x28c]
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
...

GC Heap History (10 events):
Event: 241217.875 GC heap after
Heap after GC invocations=5932 (full 1):
 par new generation   total 613440K, used 1702K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K,   0% used [0x00000006c0000000, 0x00000006c0000000, 0x00000006e1490000)
  from space 68096K,   2% used [0x00000006e1490000, 0x00000006e1639870, 0x00000006e5710000)
  to   space 68096K,   0% used [0x00000006e5710000, 0x00000006e5710000, 0x00000006e9990000)
 concurrent mark-sweep generation total 3512768K, used 1178323K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K
}
Event: 241220.784 GC heap before
{Heap before GC invocations=5932 (full 1):
 par new generation   total 613440K, used 547046K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K, 100% used [0x00000006c0000000, 0x00000006e1490000, 0x00000006e1490000)
  from space 68096K,   2% used [0x00000006e1490000, 0x00000006e1639870, 0x00000006e5710000)
  to   space 68096K,   0% used [0x00000006e5710000, 0x00000006e5710000, 0x00000006e9990000)
 concurrent mark-sweep generation total 3512768K, used 1178323K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K
Event: 241220.794 GC heap after
Heap after GC invocations=5933 (full 1):
 par new generation   total 613440K, used 2633K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K,   0% used [0x00000006c0000000, 0x00000006c0000000, 0x00000006e1490000)
  from space 68096K,   3% used [0x00000006e5710000, 0x00000006e59a2400, 0x00000006e9990000)
  to   space 68096K,   0% used [0x00000006e1490000, 0x00000006e1490000, 0x00000006e5710000)
 concurrent mark-sweep generation total 3512768K, used 1178400K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K
}
Event: 241224.818 GC heap before
{Heap before GC invocations=5933 (full 1):
 par new generation   total 613440K, used 547977K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K, 100% used [0x00000006c0000000, 0x00000006e1490000, 0x00000006e1490000)
  from space 68096K,   3% used [0x00000006e5710000, 0x00000006e59a24a8, 0x00000006e9990000)
  to   space 68096K,   0% used [0x00000006e1490000, 0x00000006e1490000, 0x00000006e5710000)
 concurrent mark-sweep generation total 3512768K, used 1178400K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K
Event: 241224.828 GC heap after
Heap after GC invocations=5934 (full 1):
 par new generation   total 613440K, used 2257K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K,   0% used [0x00000006c0000000, 0x00000006c0000000, 0x00000006e1490000)
  from space 68096K,   3% used [0x00000006e1490000, 0x00000006e16c4458, 0x00000006e5710000)
  to   space 68096K,   0% used [0x00000006e5710000, 0x00000006e5710000, 0x00000006e9990000)
 concurrent mark-sweep generation total 3512768K, used 1178467K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K
}
Event: 241224.829 GC heap before
{Heap before GC invocations=5934 (full 1):
 par new generation   total 613440K, used 8425K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K,   1% used [0x00000006c0000000, 0x00000006c0605fc8, 0x00000006e1490000)
  from space 68096K,   3% used [0x00000006e1490000, 0x00000006e16c4458, 0x00000006e5710000)
  to   space 68096K,   0% used [0x00000006e5710000, 0x00000006e5710000, 0x00000006e9990000)
 concurrent mark-sweep generation total 3512768K, used 1178467K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K
Event: 241224.836 GC heap after
Heap after GC invocations=5935 (full 1):
 par new generation   total 613440K, used 1601K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K,   0% used [0x00000006c0000000, 0x00000006c0000000, 0x00000006e1490000)
  from space 68096K,   2% used [0x00000006e5710000, 0x00000006e58a0760, 0x00000006e9990000)
  to   space 68096K,   0% used [0x00000006e1490000, 0x00000006e1490000, 0x00000006e5710000)
 concurrent mark-sweep generation total 3512768K, used 1178572K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K
}
Event: 241228.987 GC heap before
{Heap before GC invocations=5935 (full 1):
 par new generation   total 613440K, used 546945K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K, 100% used [0x00000006c0000000, 0x00000006e1490000, 0x00000006e1490000)
  from space 68096K,   2% used [0x00000006e5710000, 0x00000006e58a07a0, 0x00000006e9990000)
  to   space 68096K,   0% used [0x00000006e1490000, 0x00000006e1490000, 0x00000006e5710000)
 concurrent mark-sweep generation total 3512768K, used 1178572K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K
Event: 241228.995 GC heap after
Heap after GC invocations=5936 (full 1):
 par new generation   total 613440K, used 2411K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K,   0% used [0x00000006c0000000, 0x00000006c0000000, 0x00000006e1490000)
  from space 68096K,   3% used [0x00000006e1490000, 0x00000006e16eada0, 0x00000006e5710000)
  to   space 68096K,   0% used [0x00000006e5710000, 0x00000006e5710000, 0x00000006e9990000)
 concurrent mark-sweep generation total 3512768K, used 1178609K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K
}
Event: 241233.095 GC heap before
{Heap before GC invocations=5936 (full 1):
 par new generation   total 613440K, used 547755K [0x00000006c0000000, 0x00000006e9990000, 0x00000006e9990000)
  eden space 545344K, 100% used [0x00000006c0000000, 0x00000006e1490000, 0x00000006e1490000)
  from space 68096K,   3% used [0x00000006e1490000, 0x00000006e16eada0, 0x00000006e5710000)
  to   space 68096K,   0% used [0x00000006e5710000, 0x00000006e5710000, 0x00000006e9990000)
 concurrent mark-sweep generation total 3512768K, used 1178609K [0x00000006e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 75352K, capacity 77390K, committed 77524K, reserved 1118208K
  class space    used 7885K, capacity 8270K, committed 8296K, reserved 1048576K

Internal exceptions (10 events):
Event: 241213.639 Thread 0x00007f4b60005800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006c548d5f8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 241213.641 Thread 0x00007f4b580a5000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006c548de20) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 241213.797 Thread 0x00007f4b54917800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006c64df2a0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 241213.903 Thread 0x00007f4b5c028000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006c6bfddb8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 241222.033 Thread 0x00007f4b849e9000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006cccff568) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 241226.620 Thread 0x00007f4b849e9000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006cf3d4c60) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 241228.641 Thread 0x00007f4b60005800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006de46f1e8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 241228.642 Thread 0x00007f4b580a5000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006de46fa10) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 241228.907 Thread 0x00007f4b5c028000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006e0b591b8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 241230.221 Thread 0x00007f4b787e5000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006c9c63328) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u72/5732/hotspot/src/share/vm/prims/jni.cpp, line 735]

Events (10 events):
Event: 241232.882 loading class javax/xml/messaging/URLEndpoint done
Event: 241232.918 loading class javax/xml/messaging/URLEndpoint
Event: 241232.918 loading class javax/xml/messaging/URLEndpoint done
Event: 241232.959 loading class javax/xml/messaging/URLEndpoint
Event: 241232.959 loading class javax/xml/messaging/URLEndpoint done
Event: 241233.007 loading class javax/xml/messaging/URLEndpoint
Event: 241233.007 loading class javax/xml/messaging/URLEndpoint done
Event: 241233.050 loading class javax/xml/messaging/URLEndpoint
Event: 241233.050 loading class javax/xml/messaging/URLEndpoint done
Event: 241233.094 Executing VM operation: GenCollectForAllocation

...

Any help will be greatly appreciated.

Is this seg fault constant? If so, can you run it locally and see if it still happens? — Ermiya Eskandary, Dec 07 '22 at 10:34
This happens when it's deployed to AWS. Not happening on local. — Ankit Sharma, Dec 08 '22 at 12:42
which instance type do you use? It could be some architecture limitation on specific instance type — Rustam, Dec 09 '22 at 07:46
The JVM is for linux-amd64 - just to double check, you're not running on a Graviton CPU? Have you tried a version of Java higher than 8? — Ermiya Eskandary, Dec 12 '22 at 13:43
did u set either `Xmx` or `Xms` ? if so what's the value? also it might be because of setting docker memory limit — Mostafa Nazari, Dec 12 '22 at 15:43
Make sure the stack size is big enough for the thread, which you can control with `-Xss` — Carl Mastrangelo, Dec 13 '22 at 23:14

score 1 · Answer 1 · answered Dec 12 '22 at 16:02

it seems that the application need to allocate a large block of memory.

reading this report and this question can help.

usually there is no need to resize TLAB size but if you didn't find any solution, try tuning it by either -XX:TLABSize=N or -XX:ResizeTLAB.

if you are about to change TLAB size, it is strongly recommended to read this article, and you can take a look at this question

Segmentation fault in GC Thread

1 Answers1