jetty quits immediately after setting HEAPPROFILE

Question

I want to analyse off-heap memory usage of jetty.
So I use gperftools and add env to jetty.sh:

export LD_PRELOAD=/usr/local/lib/libtcmalloc.so
# export HEAPPROFILE=/home/cccccc/perftools/mybin

It works fine

[root@xxx ~]# lsof -n | grep tcmalloc
java      23200           root  mem       REG              253,1    2350880     667001 /usr/local/lib/libtcmalloc.so.4.5.3
java      23200 23203     root  mem       REG              253,1    2350880     667001 /usr/local/lib/libtcmalloc.so.4.5.3
java      23200 23204     root  mem       REG              253,1    2350880     667001 /usr/local/lib/libtcmalloc.so.4.5.3
java      23200 23205     root  mem       REG              253,1    2350880     667001 /usr/local/lib/libtcmalloc.so.4.5.3

But when I comment out HEAPPROFILE and try service jetty restart(start).
After logging

Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Dumping heap profile to /home/cccccc/perftools/mybin.0001.heap (Exiting, 0 bytes in use)
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
Starting Jetty: Starting tracking the heap
Starting tracking the heap
Starting tracking the heap
ok Thu Sep 20 20:33:36 CST 2018

It quits.
And check the error log, I found it's a 'libunwind' bug.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f882ad93dd4, pid=18147, tid=0x00007f882949a700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_122-b03) (build 1.8.0_122-ea-b03)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.122-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libunwind.so.8+0x2dd4]  access_mem+0x24

(the full content : https://drive.google.com/open?id=1JB3R27Udpdid3FMgnoWQPr516gZsSfc3)

too far from my scope.
I use gperftools-2.7 with libunwind-1.2.1(before I used libunwind-0.99-BETA the same result).
It may be very hard to resolve.Too many environment factors.

I try to use jemalloc instead, it works fine. – fairjm Sep 21 '18 at 04:39 — fairjm, Sep 21 '18 at 04:39

score 1 · Accepted Answer · answered Sep 22 '18 at 17:13

The issue is almost surely due to libunwind attempting to deal with stack frames for code dynamically generated by JVM (hotspot etc). I guess even "interpreter" is not really classical interpreter.

Your best bet I think is it switch to frame pointer based unwinding (pass TCMALLOC_STACKTRACE_METHOD=x86 as environment variable). And you'll need to run JVM with -XX:+PreserveFramePointer (see https://medium.com/netflix-techblog/java-in-flames-e763b3d32166)

jetty quits immediately after setting HEAPPROFILE

1 Answers1