One of our web server stopped responsding to any request, jstack suggests that there are deadlocks in the process, serval threads are stuck, the log is like this:
Found one Java-level deadlock:
=============================
"qtp728030296-143":
waiting to lock monitor 0x00007ff8e4002528 (object 0x0000000080005b28, a java.lang.ThreadGroup),
which is held by UNKNOWN_owner_addr=0x00007ff8d8002000
Java stack information for the threads listed above:
===================================================
"qtp728030296-143":
at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:865)
- waiting to lock <0x0000000080005b28> (a java.lang.ThreadGroup)
at java.lang.Thread.init(Thread.java:405)
at java.lang.Thread.init(Thread.java:349)
at java.lang.Thread.<init>(Thread.java:499)
at org.eclipse.jetty.util.thread.QueuedThreadPool.newThread(QueuedThreadPool.java:558)
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:537)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:49)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:728)
at java.lang.Thread.run(Thread.java:748)
It seems that threadgroup monitor is held by some unknown thread. I googled the error message "which is held by UNKNOWN_owner_addr", it leads me to this apache bug thread:
It has the exact same error message, and it suggests that this is caused by a JDK jemalloc bug
I'm not familiar with those low level jvm/jemalloc stuff, they are like magic to me:). So I'm not 100% percent sure if my problem is cause by this bug. And If I understand the jdk bug correctly, does it mean that all JDK8, 11, 12 programs are suffered from it, any java program can run into this mysterious deadlock with a tiny possibility?
