Jemalloc: Java Native Memory profiling shows 100% je_prof_backtrace

Question

We have a multi-threaded production Java application. We are trying to check the native memory usage as mentioned in this post.

But on the dump I am seeing 100% memory is being taken by je_prof_backtrace

ubuntu@platform1:/tmp/jemalloc_dump$ /home/ubuntu/jemalloc/bin/jeprof --show_bytes `which java` /tmp/jemalloc_dump/tsdb.62634.6454.i6454.heap
Using local file /usr/bin/java.
Using local file /tmp/jemalloc_dump/tsdb.62634.6454.i6454.heap
Welcome to jeprof!  For help, type 'help'.
(jeprof) top
Total: 815965512 B
815965512 100.0% 100.0% 815965512 100.0% je_prof_backtrace
       0   0.0% 100.0% 268451840  32.9% 0x00007f1338bd83a5
       0   0.0% 100.0% 279057735  34.2% 0x00007f13390d615a
       0   0.0% 100.0% 268455936  32.9% 0x00007f133918adda
       0   0.0% 100.0% 268455936  32.9% AllocateHeap
       0   0.0% 100.0% 268455936  32.9% JVM_MonitorWait
       0   0.0% 100.0% 268451840  32.9% Java_java_util_zip_Inflater_inflateBytesBytes
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::inflate
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::omAlloc
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::wait```

Output is like below if I use text

(jeprof) text
Total: 815965512 B
815965512 100.0% 100.0% 815965512 100.0% je_prof_backtrace
       0   0.0% 100.0% 268451840  32.9% 0x00007f1338bd83a5
       0   0.0% 100.0% 279057735  34.2% 0x00007f13390d615a
       0   0.0% 100.0% 268455936  32.9% 0x00007f133918adda
       0   0.0% 100.0% 268455936  32.9% AllocateHeap
       0   0.0% 100.0% 268455936  32.9% JVM_MonitorWait
       0   0.0% 100.0% 268451840  32.9% Java_java_util_zip_Inflater_inflateBytesBytes
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::inflate
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::omAlloc
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::wait

Steps followed.

Downloaded jemalloc 5.2.1 release tag from here.
./autogen.sh --enable-prof inside the directory where jmalloc tar in untared.
make

Application was started with the below options

export JEMALLOC_PATH=/home/ubuntu/jemalloc
export MALLOC_CONF=prof:true,lg_prof_interval:21,lg_prof_sample:28,prof_prefix:/tmp/jemalloc_dump/tsdb
LD_PRELOAD=${JEMALLOC_PATH}/lib/libjemalloc.so.2 \
java ...

I am new to jemalloc. Can someone suggest what could be going wrong?

ubuntu@platform1:/tmp/jemalloc_dump$ java -version
openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment Zulu11.48+22-SA (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.48+22-SA (build 11.0.11+9-LTS, mixed mode)

OS - Ubuntu 18.04

The Java heap is not the same the heap provided by any `malloc()` implementation. — Andrew Henle, Jun 21 '21 at 13:40
I know. Somone edited the title. I am looking to check native memory usage. — tuk, Jun 21 '21 at 13:46
According to [this bug](https://github.com/jemalloc/jemalloc/issues/507) this is not abnormal. Do you see stacks if you issue `text` instead of `top`? — Botje, Jun 21 '21 at 14:01
@Botje I am seeing some hexadecimal number if I use `text`. I have updated the question with this info. — tuk, Jun 21 '21 at 14:12
Those appear to be the stack addresses of your Java threads. I'm a bit confused it is not segregating the dump by thread.. — Botje, Jun 21 '21 at 14:15
@Botje Is there some issue in the version of jemalloc (5.2.1) I am using? — tuk, Jun 21 '21 at 14:26

score 1 · Accepted Answer · answered Jul 08 '21 at 12:00

1

Try configuring using the below flags :

./configure --enable-prof --enable-stats --enable-debug --enable-fill

answered Jul 08 '21 at 12:00

Pranjul Ahuja

26
1
3

score 0 · Answer 2 · answered Oct 27 '21 at 07:02

This has been discussed in github also. Posting the relevant reply

Quoting Jason from the original thread:

This is usually caused by the backtracing facility failing to do anything beyond reporting the caller's return instruction pointer. You may have better luck if you use libunwind, but unfortunately I've seen all the supported backtracing mechanisms fail under various conditions.

In other words, if you haven't tried libunwind, give that a try. The reported issue happens when the stack unwinder cannot walk the stack, in which case jemalloc / jeprof doesn't have any backtrace data to report.

Closing because nothing actionable from the jemalloc side.

Jemalloc: Java Native Memory profiling shows 100% je_prof_backtrace

2 Answers2