3

Not really being knowledgeable about Java and especially about debugging in Java, but taking a heap dump in Jenkins using Monitoring and then decoding it in Eclipse with MAT shows total memory used 169.4 MB, while in Jenkins monitoring the memory seems to be constantly used a lot and GCs are running frequently. -XmX is 4G.

How come I only get 169.4 MB with MAT? Could it be because before making a dump Jenkins executes a GC? If so, can I avoid it to see the full memory dump?

Zloj
  • 2,235
  • 2
  • 18
  • 28
  • AFAIK a dump will cause a gc before or at least MAT will ignore what could be gc'ed anyways. Jenkins will probably load a lot of data during a job which is eligible for gc afterwards. Btw, how do you trigger the dump? Is it a built-in function of Jenkins or are you sending a command to the JVM via JConsole or similar? – Thomas Sep 25 '15 at 10:40
  • @Thomas dump can be triggered from monitoring plugin. There's a button - that's all I can say :) That comes from Jenkins GUI – Zloj Sep 25 '15 at 10:42
  • In that case it could be that the plugin stops the current job or waits for it to finish (or there's just a lot of data the current job loaded but doesn't need anymore). - I'm just guessing here since I don't know that plugin. But besides that, do you have issues with frequent GCs? – Thomas Sep 25 '15 at 10:44
  • @Thomas I don't think so. With 4G memory GC is executed once in around 30 seconds and takes 1 second. I have problems with slow loading pages / console logs, but no network issues are present and Jenkins log says nothing. No errors anywhere, just pages being loaded slowly. Even system load on master node rarely raises above 1 ( running 64 bit RHEL with 4 cores and 24 G ram alongside other 6 Jenkins instances which use less memory). – Zloj Sep 25 '15 at 10:48
  • @Thomas that being said - executing GC before collecting a dump ... Isn't this a bit wrong? How would you troubleshoot the memory problems if something reserves too much memory? It just gets cleaned up so one never finds out... Gonna try to find some console commands for heap dump without GC. – Zloj Sep 25 '15 at 10:54
  • Well you'd normally have memory problems which you need a dump for when memory is filled up by objects that _can't_ be gc'ed. Thus you'd normally not want to see objects which can be collected anyways. – Thomas Sep 25 '15 at 11:01
  • What does the report say about "Used non heap memory"? (two Thomases now) – Thomas Weller Sep 25 '15 at 11:22
  • @Thomas I can't see this in any of MAT's report. Where can I take a look at "Used non heap memory"? The file has .hprof extension which I assume should indicate that it is heap-only. I haven't taken a dump without GC yet. – Zloj Sep 25 '15 at 11:42
  • I've now taken a dump using jmap. It also seems to execute GC before collecting heap dump, so I can't take an in-depth look into what uses memory the most with this either. Problem #1 of modern software: no one cares about memory as long as there are no leaks and it isn't a toster with a small chip or space station. Thank you for help Thomas #1 and #2 :) – Zloj Sep 25 '15 at 12:07
  • With "Used non heap memory" I was referring to [this graphics](https://wiki.jenkins-ci.org/download/thumbnails/41877877/graphics.png). It will not be part of the dump. – Thomas Weller Sep 25 '15 at 12:13
  • @Thomas it says it uses strictly between 1.5 and 3.5 G. I assume this is a bit of a lie, since looking at 'currently used memory' in monitoring I can see that it acts as an equalizer node, though a bit slow - going from 0 to a bit less than max and dropping again. – Zloj Sep 25 '15 at 12:30
  • Just a minor clarification: Not always the decrease of used memory drops to 0. I don't have the time to make detailed statistics on this, but continuous FULL GC isn't the problem here. – Zloj Sep 25 '15 at 12:41

3 Answers3

3

Understanding memory

Yes, a Java heap dump and a virtual memory dump (called "crash dump" or "memory dump" on Windows) are different things.

The Java heap dump only contains the memory relevant for Java, i.e. the place where Java objects reside. A Java heap dump is analyzed with tools like MAT (as mentioned by you) or the Java Heap Analysis Tool

A Windows crash dump of a (user mode) process contains all virtual memory, where virtual memory is the term of the operating system providing the memory. On Windows, that's all memory allocated via VirtualAlloc.

The OS virtual memory will include the Java heap, since Java can only request memory from the operating system.

So, when comparing memory sizes, it's important to understand whether the tool is Java specific or OS general.

In your case, Monitoring looks much like a generic tool, since it deals with a process list and CPU times, nothing which seems to be Java specific. On the other side, MAT is clearly a Java tool from its description.

Memory differences

So how much can the Java heap size differ from the virtual memory size?

Much:

  1. Loaded EXEs/DLLs account to the virtual memory, but not to the Java heap
  2. Memory used by native code (e.g. via JNI) accounts to virtual memory, but not to the Java heap
  3. Memory requested by Java from the OS, but not used by Java yet, definitely accounts as virtual memory, but could be reported as "free" from Java perspective.
Thomas Weller
  • 55,411
  • 20
  • 125
  • 222
  • Thanks for a good answer. Just one minor correction: in this case `monitoring` is actually monitoring Jenkins JVM and not the operating system. – Zloj Sep 30 '15 at 19:18
1

Apparently, the tools that collect heap dump execute GC to reduce the size of the dump. Since things that can be GCed shouldn't produce the OOM this is intended rather to find memory leaks than troubleshoot memory usage.

Zloj
  • 2,235
  • 2
  • 18
  • 28
0

VM requests virtual memory to store variety of data. VM then allocates some memory to store variables (that is heap), native code (not heap), reserves some memory that is not used yet. As a result virtual memory is strictly larger than heap. There isn't any clear correlation between heap and virtual memory given that you can write a simple infinite recursion (heap will be almost as large as virtual memory) or load a large dll in a hello world program (heap is much smaller than virtual memory).

sixtytrees
  • 1,156
  • 1
  • 10
  • 25