We have a microservices environment with two servers and 50 java(11) apps managed by tomcat. Each app have small heap (no more than 2 GB) and starts with 500 MB (-Xms).
Sometimes, after reboot the server, we observe that G1 GC behaviour change, and (after 30 hours) goes from 5000 GC starts to 400 GC starts with exactly the same numer of requests, so heap is a bit larger and no memory is returned to OS. This is a problem with OS memory when this behaviour take place with 50 apps.
Foro example this is a VM.info for Server_A and App_A:
Event: 118110.566 GC heap after {Heap after GC invocations=4884 (full 0): garbage-first heap total 294912K, used 113497K [0x00000000c0000000, 0x0000000100000000) region size 1024K, 1 young (1024K), 1 survivors (1024K) Metaspace used 184030K, capacity 188339K, committed 189344K, reserved 1216512K class space used 21086K, capacity 22563K, committed 22860K, reserved 1048576K }
And the same statistics for Server_B and App_A.
Event: 111212.036 GC heap after {Heap after GC invocations=344 (full 0): garbage-first heap total 891904K, used 175855K [0x00000000c0000000, 0x0000000100000000) region size 1024K, 12 young (12288K), 12 survivors (12288K) Metaspace used 186664K, capacity 192522K, committed 192776K, reserved 1218560K class space used 21741K, capacity 23879K, committed 23936K, reserved 1048576K }
In Server_A the heap it's 294912K with 4884 GC invocations
In Server_B the heap it's 891904K with 344 GC invocations.
Same behaviour for the rest of java apps in Server_B
Server_A and Server_B are clones. Sometimes reboot Server_B solves the problem and G1 GC take place at regular intervals. Sometimes reboot Server_A and problems with "GC starts" take place again.
How is possible this behaviour?
Any suggestion?
Thanks a lot.
We expect the same GC behaviour each time we reboot the server.
