Why all java utils for dumping heap are not working? How to dump heap?

Question

I'm tired to trying to dump java heap when I need it. Any time when I could do that none of utils are not able to do that. I'm trying jmap, jvisualvm, jconsole, jmc and none of them not able to do that easy task -- just dump the *** heap!

How people uses that programs???

First, I tried visual instruments, but none of them have worked (sorry, but I didn't think that I'll need screenshots of their errors). Ok, I did think, but console tools should work!

$ jmap -dump:format=b,file=server.bin 136447
Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /proc/136447/root/tmp/.java_pid136447: target process 136447 doesn't respond within 10500ms or HotSpot VM not loaded
    at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:103)
    at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
    at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
    at jdk.jcmd/sun.tools.jmap.JMap.executeCommandForPid(JMap.java:128)
    at jdk.jcmd/sun.tools.jmap.JMap.dump(JMap.java:218)
    at jdk.jcmd/sun.tools.jmap.JMap.main(JMap.java:114)

What?

$ jmap -F -dump:format=b,file=server.bin 136447
Error: -F option used
Cannot connect to core dump or remote debug server. Use jhsdb jmap instead

Ok...

$ jhsdb jmap --binaryheap --dumpfile server.bin --pid 136447
Attaching to process ID 136447, please wait...
ERROR: ptrace(PTRACE_ATTACH, ..) failed for 136447: Операция не позволена
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 136447: Операция не позволена
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 136447: Операция не позволена
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:179)
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:337)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:672)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:612)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:338)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:187)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:321)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:406)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 136447: Операция не позволена
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:328)
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:154)

Hm...

$ sudo jhsdb jmap --binaryheap --dumpfile server.bin --pid 136447
[sudo] пароль для Mingun: 
Attaching to process ID 136447, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 14.0.2+12-Ubuntu-120.04
sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007f20500286d0
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
    at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
    at jdk.hotspot.agent/sun.jvm.hotspot.memory.Universe.heap(Universe.java:102)
    at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:423)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:262)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:225)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:321)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:406)

WTF???

This is the basics! Why the java tools are so poor? How I should get the heap? That is not a single case. I've got such results every time when I trying to get the dump

For the reference, this is how program launched:

java -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -XX:+PrintConcurrentLocks -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9009 -server -Xmx3G -Duser.language=en -Duser.region=US <main class> <app parameters>

This reads more as a rant than a question. Please edit your question and remove the ranting. — Mark Rotteveel, May 13 '22 at 09:25
I am not sure this is "basics": in my experience maybe 10-20 out of 100 Java developers know about these tools, and maybe 3-5 use them. Therefore these tools seem to be more marginal than the rest of the Java ecosystem. There are implementation details that could be polished out if this toolset was more mainstream; on the other hand, there are objective limitations coming from the OS etc. I actually think these tools are pretty good compared to other platforms, but maybe I am not aware of the alternatives. What is the platform you have in mind where this would be easy? — starikoff, May 13 '22 at 09:25
Basics because everyone recommends you to use them when you have troubles, but they just not working — Mingun, May 13 '22 at 11:36

starikoff · Answer 1 · 2022-05-13T10:15:47.147

2

jmap -dump:format=b,file=server.bin ...: will only work if 1) the JVM is not frozen, e.g. due to permanent full GC, because it uses a cooperative mechanism where the target JVM actually runs commands doing the heap dump (Java Attach API), 2) you run jmap as the same user as the one who started the JVM. I take from your comments that condition 1) is not met in your case.
jmap -F -dump:format=b,file=server.bin ...: the option -F was removed in JDK 9.
jhsdb jmap --binaryheap --dumpfile server.bin --pid ...: fails because you either need to run it as root or set some liberal restrictions in the linux kernel for ptrace (ptrace_scope should be 0).
sudo jhsdb jmap --binaryheap --dumpfile server.bin --pid ...: has a restriction that jhsdb must come from the exactly same JDK version as the target JVM (like, if the target JVM has version 14.1.2+3 then I would make sure that jhsdb comes from JDK 14.1.2+3 as well just for safety). Seems to be the case in your case because no warning is printed, but I would suggest to check it for sure. The only other thing I can think of is you might need to install openjdk-debuginfo as mentioned e.g. here.

edited May 13 '22 at 10:15

answered May 13 '22 at 09:16

starikoff

1,601
19
23

@Mingun By solving problems with the 4th command you tried and using it. I gave 2 suggestions, if they do not help and you solve this differently, I will be happy to remove my answer. – starikoff May 13 '22 at 11:31
1. Ok, so how I should get a dump when JVM is frozen? 2. Ok, at least it is not mentioned in the `jmap --help`, but error that was shown is confusing 3. Strange, that this is not mentionen in the help 4. `jhsdb` from the same distribution as `java`: ``` $ ll /etc/alternatives/jhsdb lrwxrwxrwx 1 root root 44 дек 18 2020 /etc/alternatives/jhsdb -> /usr/lib/jvm/java-14-openjdk-amd64/bin/jhsdb* $ ll /etc/alternatives/java lrwxrwxrwx 1 root root 43 дек 18 2020 /etc/alternatives/java -> /usr/lib/jvm/java-14-openjdk-amd64/bin/java* ``` `/usr/bin/xxx` links to the `/etc/alternatives/xxx` – Mingun May 13 '22 at 11:34
Please keep your answer, it gives very useful info. I don't think that the problem in debug symbols, because I cat get a heapdump when app is idle, but it is not interesting. I need a heap when app in troubles and it seems that those instruments just does not able to work in that conditions – Mingun May 13 '22 at 11:43
@Mingun Just to clarify: you can get the heap dump when the app is not frozen using `jhsdb` or `jmap`? Because if it's `jmap`, it is expected to be more resilient with a non-frozen JVM. Make sure `jhsdb` works under non-frozen conditions. – starikoff May 13 '22 at 11:57
yes, I think so. Actually, in that particular case I was able to take a dump, but I think that it is because the app finally fallen into OoM, so `jmap` was able to do its work, since CPU consumption stopped for a few seconds around this point. I cannot say exactly what leaded to this. – Mingun May 13 '22 at 14:36

score 0 · Answer 2 · answered May 13 '22 at 07:08

0

Firstly you need find the real java jpid:

launch jps, find the pid value like:

./jps
15552 org.eclipse.equinox.launcher_1.6.400.v20210924-0641.jar
2816 org.eclipse.equinox.launcher_1.6.400.v20210924-0641.jar
29504 org.eclipse.equinox.launcher_1.6.400.v20210924-0641.jar
34032 Jps
39616 org.eclipse.equinox.launcher_1.6.100.v20201223-0822.jar
10248 org.eclipse.equinox.launcher_1.3.201.v20161025-1711.jar

Your pid should be one of the list. Please find yours.

create the dump

./jmap -dump:live,format=b,file=heap.bin 39616

Finally you will find the heap.bin

answered May 13 '22 at 07:08

SeanH

538
2
8

The PID is correct -- I can attach to the process when program runs without problems – Mingun May 13 '22 at 08:58
Oh. Maybe it's related with the jvm version you are using. Did you try the launch cmd without: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9009 ? – SeanH May 13 '22 at 09:13

score -1 · Answer 3 · answered May 13 '22 at 08:37

-1

What's your issue?

Just read the **** manual, search it for the error message/the exception: "com.sun.tools.attach.AttachNotSupportedException".

Based on that I would assume that the JVM, that you query for a heap dump, is configured in a way that it does not support the Attach API, and therefore all the console tools are rendered useless …

Usually, this is the default (or had been the default) for J9 JVMs, but from the provided stacktraces, at least your tools are running on Hotspot.

answered May 13 '22 at 08:37

tquadrat

3,033
1
16
29

That is not true, because I can attach to the process while it runs without problems, but when it starts to consume CPU due to GC work, none of instruments could give me any useful information when I needed it – Mingun May 13 '22 at 08:57

Why all java utils for dumping heap are not working? How to dump heap?

3 Answers3