7

I am having an interesting dilemma where I appear to have a memory leak (or growing data structure). I get the typical "raising linearly over time" graph when I analyse my memory usage. Trying to figure out what the cause of the problem is, I did a heap-dump. What I found was that over 50% of memory was being allocated for a ConcurrentLinkedQueue node. The top consumers of memory are, com.singularity.ee.agent.util.ch and java.util.concurrent.ConcurrentLinkedQueue$Node as seen in the picture below.

enter image description here

I dont know what a util.ch is but it appears to be tied to the Node, as each ch has an immediate reference to a node, so no worries focusing on that.

Now trying to find references to the closest GC for a $Node, I get the following:

enter image description here

Whats strange about this is that it does not have a ConcurrentLinkedQueue$Node, or even a ConcurrentLinkedQueue as a parent at all. All references are strange types that I do not understand, kh, uc, z, g, etc. Does anyone know what these types are?

I am trying to find out what exactly is causing the problem, but I cant seam to find how these nodes are even being created/saved.

Here is the kicker: I do not use the ConcurrentLinkedQueue anywhere in my code. I do use a ConcurrentHashMap, but there are not very many HashMap$Node's so that shouldnt be the issue.

Does anyone have any ideas as to how these nodes are being created or why I have so many instances of them?

To answer dependency questions: I am running tomcat 6, java 6, Java Spring.

Nick Humrich
  • 14,905
  • 8
  • 62
  • 85
  • 1
    the class name sound like obfuscated code and this is used to make it harder to analyze the code. This is done either on sold frameworks or by attackers. As singularity.com doesn't look like a software vondor, I'd be careful if I were in your shoes. And I can't help you much besides a small debugging trick: getClass().getProtectionDomain().getCodeSource() helps a lot to figure out where the classes are loaded from. This might be useful to figure out what it is and how to get rid of it. Good luck! – Bernd Ebertz Apr 24 '14 at 18:29
  • On a more thorough google search, com.singularity.ee.agent appears to be linked to AppDynamics (though I cant find the exact class i am using here), which I am using to monitor memory. Perhaps AppDynamics is causing the problem itself. – Nick Humrich Apr 24 '14 at 18:59
  • @Humdinger What IDE are you using? If you remove AppDynamics and instead use MAT (in Eclipse) https://www.eclipse.org/mat/ do you get the same results? – durron597 Apr 24 '14 at 19:51
  • @durron597 I am using IntelliJ. I can't exactly "remove AppDynamics" because this is on a production server. I have opened a ticket with AppDynamics however and I will see what they say. I am just analysing the heap dump from the server with VisualVM because I cant duplicate the memory leak locally. – Nick Humrich Apr 25 '14 at 15:40
  • What classes contain the references to `.util.ch`? Can you keep jumping up the chain until you get to a class that isn't obfuscated? – durron597 Apr 25 '14 at 15:58
  • @durron597 the `ConcurrentLinkedQueue$Node` is what contains a reference to the `.util.ch` so the reference stack looks almost the same (except sometimes there is a chain of nodes and ch's). Let me know if you want to see it. – Nick Humrich Apr 25 '14 at 16:04

2 Answers2

2

Turns out I had some proprietary code from AppDynamics causing this issue. I opened a ticket with them and they fixed the issue in their next release. Thank you for all the help though!

Nick Humrich
  • 14,905
  • 8
  • 62
  • 85
0

From your comments, since its the linked list that's referencing the *.util.ch class, you can look for jars that have a dependency on the ConcurrentLinkedQueue along with the *.util.ch class.

Once the jar is identified, then depending upon whether the containing class was built in-house or if a third party jar you can check if the leak can be fixed or if a fix is available.

Regarding how to determine dependency, see if this helps - Find all dependencies in a Java class

Unless reflection is used, the class (and therefore containing jar) should be easy to find. If there is a possibility of reflection being used, then extract the jars and try doing a 'contains' search on all the *.class files.

PS : If the solution has already been found, do indicate as such!

Community
  • 1
  • 1
Ravindra HV
  • 2,558
  • 1
  • 17
  • 26