JVM crash when java.nio.DirectByteBuffer trying to free memory in GC

Question

When I was running a Apache Spark job, which few lines of input data, the executor JVM crashed due to free java.nio.DirectByteBuffer in the GC:

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  sun.misc.Unsafe.freeMemory(J)V+0
j  java.nio.DirectByteBuffer$Deallocator.run()V+17
J 1537 C1 java.lang.ref.Reference.tryHandlePending(Z)Z (115 bytes) @ 0x00007f082d519f94 [0x00007f082d5199c0+0x5d4]
j  java.lang.ref.Reference$ReferenceHandler.run()V+1
v  ~StubRoutines::call_stub

And there was no memory pressure:

Heap:
par new generation   total 153344K, used 17415K [0x0000000738000000, 0x0000000742660000, 0x0000000742660000)
eden space 136320K,   1% used [0x0000000738000000, 0x00000007381955c8, 0x0000000740520000)
from space 17024K,   92% used [0x0000000740520000, 0x000000074148c778, 0x00000007415c0000)
to   space 17024K,    0% used [0x00000007415c0000, 0x00000007415c0000, 0x0000000742660000)
concurrent mark-sweep generation
total 2057856K, used 76674K [0x0000000742660000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace       used 49890K, capacity 50454K, committed 50540K, reserved 1093632K
class space     used 6821K, capacity 6995K, committed 7056K, reserved 1048576K

Full hs_err file: http://www.evernote.com/l/AAQu5abObUND5KFJbFNO9RpVfLQlBiwX6gg/

I'm voting to close this question as off-topic because Stack Overflow is not a debugging service. — user253751, Sep 20 '16 at 22:17
The is most like a bug either in the JVM or Apache Spark. The error indicates it was trying to free something which wasn't allocated in the first place. — Peter Lawrey, Sep 20 '16 at 22:23
or a double free, which could be caused by a race between something using internal APIs to release DBBs and the reference handler. — the8472, Sep 20 '16 at 23:14
Good guess, is there a way to see where these `DirectByteBuffer` come from ? — gpcuster, Sep 21 '16 at 00:48

score 0 · Answer 1 · answered Oct 25 '16 at 06:48

You tried to play around with memory using sun.misc.Unsafe api's that resulted in segmentation fault, hence jvm crashed.

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)

j sun.misc.Unsafe.freeMemory(J)V+0 j java.nio.DirectByteBuffer$Deallocator.run()V+17 J 1537 C1 java.lang.ref.Reference.tryHandlePending(Z)Z (115 bytes) @ 0x00007f082d519f94 [0x00007f082d5199c0+0x5d4] j java.lang.ref.Reference$ReferenceHandler.run()V+1 v ~StubRoutines::call_stub`

Yes, that's because the data structure I was trying to shuffle to reducer side used `DirectByteBuffer`. — gpcuster, Oct 25 '16 at 19:30

score -1 · Answer 2 · answered Sep 27 '16 at 08:13

-1

With @the8472 's suggestion, I tried BTrace and figured out the DirectByteBuffer are comming from default Kyro serializer. So I added my own Kyro serializer to deal with avro data and everything is fine now.

answered Sep 27 '16 at 08:13

gpcuster

29
1
1
4

JVM crash when java.nio.DirectByteBuffer trying to free memory in GC

2 Answers2