-1

In our team, we are using a service which has spill over problem. It is caused by long API latency in which GC time took most of the parts. Then I found that the heap memory usage is very high. I got the heap dump using jmap for the service which is about 4.4 GB. I used the Eclipse Memory Analyzer to parse the heap dump. I found that 2.8GB of the heap dump is unreachable objects. Anyone has the suggestions that what should I do to further debug this problem? Thank you.

user7503314
  • 21
  • 1
  • 5
  • Have you tried to find out what these unreachable objects are and where they come from? – fvu Feb 08 '17 at 00:01
  • The major problems that I faced with memory leaks was with IO Streams. Check if you haven't forgot to closing them. Note: they stay on memory even if you set the variable to null (if you don't close, of course) – D.Kastier Feb 08 '17 at 00:02
  • The problem is I don't know how to check where the unreachable objects come from coz they are unreachable. Could you plz give me some suggestions? – user7503314 Feb 08 '17 at 00:26

1 Answers1

0

If you have a heap dump from when it run out of memory, I suggest use MAT to find any suspicious dominator trees which is narrow reference path to a large retained set size.

It could be same classes are ending up in different class loaders or could be HTTP session retention if web application or bad cache problem.

I suggest you, start with simple things first.

  1. Quick look at what and where jars are being loaded.
  2. Make sure class unloading is enabled (with CMS).
  3. Use memory analyser tool on the heap dump to see exactly what is being retained and by whom.
fabfas
  • 2,200
  • 1
  • 21
  • 21
  • 1. https://www.amazon.com/photos/share/25az5AbCzePBzR9AWU8TJU17ionXej2c49cFjjohClF 2. https://www.amazon.com/photos/share/8Dib5KWnPAY5y4BkeryDY4hcA0GLh2xTHewE3OjBixT I used the MAT to analyze the hprof file. But as I said, MAT skips a lot of the unreachable objects. The unreachable objects took almost 3 GB as the second picture shows. I think the flag CMSClassUnloadingEnabled will help, but I have a question that is the CMSClassUnloadingEnabled enabled by default in JVM 1.8? I'm also trying to use visualVM to analyze the hprof file. – user7503314 Feb 08 '17 at 08:19
  • You can use `-XX:+PrintFlagsFinal` option to display what options HotSpot ended up using for running Java code while `-XX:+PrintFlagsInitial` display what options were provided to HotSpot initially. e.g. `java -XX:+PrintFlagsFinal -version | grep -iE 'CMSClassUnloadingEnabled'` will tell you whether it is enabled or disabled. – fabfas Feb 08 '17 at 09:53
  • Dominator Tree view will list of each object instance and the reference tree structure relation. You can find the maximum amount of memory occupied several objects in percentage. Just a hint, you may spot something! – fabfas Feb 08 '17 at 10:13
  • Thank you so much. I checked the java -XX:+PrintFlagsFinal -version | grep -iE 'CMSClassUnloadingEnabled'. and it shows that bool CMSClassUnloadingEnabled = true {product} – user7503314 Feb 08 '17 at 19:31
  • It seems that the CMSClassUnloadingEnabled have been enabled. – user7503314 Feb 08 '17 at 19:31