Java jmap -heap command gave OutOfMemoryError

Question

We are periodically runnign jmap -heap command to monitor a tomcat application. However this is first time when we got OutOfMemoryError.

When we monitor the memory at server level there is around 110MB available. Since it is just monitoring can we ignore this time assuming no harm has been done to the application and the server was low on memory when this command run?

$ ./jmap -heap 13511
Attaching to process ID 13511, please wait...
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.tools.jmap.JMap.runTool(JMap.java:179)
        at sun.tools.jmap.JMap.main(JMap.java:110)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:597)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.<init>(LinuxDebuggerLocal.java:210)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.setupDebuggerLinux(BugSpotAgent.java:816)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.setupDebugger(BugSpotAgent.java:518)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
        at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:39)

UPDATE: We tried to run jmap -heap again and got java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.

So, the problem seems to be at OS resource level.

$ ./jmap -heap 13511
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.
#
#  Internal Error (gcTaskThread.cpp:38), pid=13162, tid=1104300352
#  Error: Cannot create GC thread. Out of system resources.
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
# An error report file with more information is saved as:
# /data01/home/s618199/JAVA/jdk1.6.0_18/bin/hs_err_pid13162.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#
Aborted

Your application will go down if you get OOM so there is a big risk running jmap to produce heap dump periodically. -XX:+HeapDumpOnOutOfMemoryError is better option to debug OOMs — Sameer Naik, Sep 16 '15 at 07:11
@SameerNaik: We only want to monitor memory and don't want to produce heap dump. I didn't know if jmap -heap option produces heap dump. — Parvez, Sep 16 '15 at 07:22
See this. http://www.herongyang.com/Java-Tools/jstack-jmap-Generate-Heap-Dump-File.html It does produce heap dump. To monitor memory, you could use jmx based tools like jconsole. — Sameer Naik, Sep 16 '15 at 07:26
@SameerNaik: Thanks. I thought not all options of jmap produces heap dump. I will research and stop using jmap for memory monitoring if -heap option also produces heap dump. We cannot use jconsole or jvisualm for monitor due to production environment limitation. — Parvez, Sep 16 '15 at 07:36
Based on my research only jmap -dump option produces the heap dump. Reference: http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html#options — Parvez, Sep 16 '15 at 07:54

score 2 · Accepted Answer · answered Sep 16 '15 at 07:12

2

This is not a memory problem (even though it is reported as one) but an operating system resource problem: You can not create any more threads. And it seems to affect the jmap program and not Tomcat.

answered Sep 16 '15 at 07:12

Codo

75,595
17
168
206

Thanks. You are right. We cannot login to the server using putty session now. We are getting "-bash: fork: Resource temporarily unavailable". – Parvez Sep 16 '15 at 07:25
Would you know if jmap -heap option produces heap dump or just analyze the memory of process and print the status? – Parvez Sep 16 '15 at 07:38
It think it just prints a summary and status. Only `jmap -dump` creates a heap dump. – Codo Sep 16 '15 at 08:03
Yes, that's correct. Thanks a lot. I already added the comment to question. Reference: http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html#options – Parvez Sep 16 '15 at 08:28

Java jmap -heap command gave OutOfMemoryError

1 Answers1