0

I'm about to start using a 3rd party closed-source library to load a bunch of data and wanted to check how fast it is and how much memory it requires to load one 'set' of data. I wrote this simple harness to invoke a data load to time it and used YourKit to have a quick look at memory usage and delve in to the CPU time.

I'm running it on Windows 7, using Eclipse on JDK8 with no VM args.

public static void main(String[] args){
    long start = System.currentTimeMillis();

    // There are a few more calls involved, but not much
    BlackBoxDataProvider bd = new BlackBoxDataProvider("c:\\thedata");
    BlackBoxData = bd.loadTheData();
    System.out.println(System.currentTimeMillis() - start + "ms");

    // Keep the application alive so I can have a quick look at memory usage
    while(true) {
        Thread.sleep(1000);
    }
}

Here's the YourKit snapshot of memory after the load is complete:

enter image description here

I then used YourKit to "Force" Garbage Collection and this happened:

enter image description here

Obviously it's not a real life scenario because I'm stuck inside the main method, on the main thread, so some of my references won't be cleaned up, but I can't figure out why the memory allocation would keep increasing.

Every time I click 'Force System GC', the allocation increases. I got up to 11.9GB before it stopped increasing.

enter image description here

Why is this happening?

beirtipol
  • 823
  • 5
  • 21
  • From what I can see, the heap behaves as expected, i.e. the Eden space and Old Gen do not participate in that growth. Only that “Allocated All Pools” is growing… – Holger Sep 18 '18 at 17:04
  • Yes, my question is why the 'allocated all pools' is growing continuously. – beirtipol Sep 19 '18 at 10:31
  • That boils down to the question what “Allocated All Pools” actually means, which only YourKit can answer. Is it MetaSpace, off-heap memory (I/O buffers), or something else? So, did you try with a different monitoring tool (e.g. VisualVM or FlightRecorder) and compare the result? – Holger Sep 19 '18 at 10:47
  • Looking with ProcessExplorer shows the private bytes growing at the same rate as 'allocated all pools'. The process continues to allocate more memory, presumably for the GC, and I'd like to know why it's allocating so much – beirtipol Sep 19 '18 at 11:37
  • I didn’t doubt that it is allocating bytes, but it would be helpful to know *why* or for what purpose. As said, this memory doesn’t belong to the ordinary heap, so it would be helpful to know the actual category. Other tools like the mentioned VisualVM or FlightRecorder might provide more helpful statistics. Also, trying with other tools helps verifying that the memory allocation is not caused by YourKit itself. You may also replace `while(true) { Thread.sleep(1000); }` with `while(true) { Thread.sleep(1000); System.gc(); }` to see whether GC triggered by the application has similar behavior. – Holger Sep 19 '18 at 12:12

1 Answers1

0

The System.gc() will return when all objects have been scanned once. If objects implementing the finalize() method are added to a queue, to be cleaned up later. This means those objects cannot be cleaned up yet (not the queue nodes which hold them) i.e. the act of triggering a GC can increase memory consumption temporarily. This is what might be happening in your case.

In short, not all objects can be cleaned up in one cycle.

Vinnie
  • 452
  • 5
  • 24
  • What if the heap itself gets filled up and there is a need for Full GC. What space does GC thread use in that case, reserved space? –  Sep 18 '18 at 15:14
  • That makes sense for the first call, but does it keep duplicating the queue of references for cleanup? Used memory isn't increasing, just the allocation the JVM is grabbing. – beirtipol Sep 18 '18 at 15:38
  • @AvinashReddyPaduri If the heap completely full then, you will get a OutOfMemoryError – Vinnie Sep 18 '18 at 15:53
  • @beirtipol There is a finalization queue it adds to. The more amount of times you call gc the more objects gets added to the queue. I recommend reading on [Java Hotspot Garbage collection](http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140228.html) – Vinnie Sep 18 '18 at 16:03
  • @Vineeth, I know it will generate OutOfMemoryError when heap gets full and not able to store any more objects. That wasn't my question, a Full GC cleans both yound and old generation spaces. If I am not worg, a Full GC should occur when heap is about to full or is full, if GC stores it's queue in heap then where will allocate the space? Not only in that case, whenever GC is used, what part of the memory is used in heap (does GC thread use heap)? If yes, is it in reserved space of the heap? –  Sep 18 '18 at 17:23
  • If GC uses heap to clean generational spaces, and if heap is full, there shouldn't be any space for the so called queue of GC to get stored on heap. Then how will GC clean the heap? That is why I have asked where does memory allocated for GC thread's objects, if it is in heap is it in reserved space (I guess that should be the only space left other than young and old generation though reserved is part of both generations, PermGen is removed from Java8 anyway). –  Sep 18 '18 at 17:27
  • @AvinashReddyPaduri I am not sure if the GC queue is stored in the heap. But it's an interesting question. Will post the answer if when I find the answer. – Vinnie Sep 18 '18 at 19:29
  • 1
    The GC will not only enqueue objects needing finalization but also [reference objects](https://docs.oracle.com/javase/8/docs/api/?java/lang/ref/Reference.html) for weakly, softly and phantom references. On HotSpot/OpenJDK, the objects with a non-trivial `finalize()` method will get a [special reference](https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/lang/ref/Finalizer.java) object not visible in the API, which is already created when the object is created. So, the GC only links these objects using a field within them, so no allocation is needed. – Holger Sep 19 '18 at 06:31
  • @Vineeth - I've read that paper on the hotspot collector, but that doesn't explain to me why the allocated memory grows to 12-20x the maximum heap size reached after System.gc() is called – beirtipol Sep 19 '18 at 10:33