10

After upgrading our Spring web app maven build to Java 11 we see a constant increase in memory consumption from the java process.

Works fine: Build with Java 8 JDK + run on server with Java 11

Has leak: Build with Java 11 + run on server with Java 11

The leak is not visible in a heap dump or even Native Memory Tracking, the process keeps increasing until physical memory + swap is full and the process is killed by the system. What kind of issue could even be causing this kind of problem?

trincot
  • 317,000
  • 35
  • 244
  • 286
Jeppz
  • 912
  • 1
  • 8
  • 31
  • 2
    Sorry for asking, but you have checked that it is not the Metaspace that is growing indefinitely and finally caused the JVM to be killed by the system? The Metaspace is holding the loaded classes and it is not part of the Heap. – tquadrat May 18 '20 at 13:30
  • @tquadrat Yea we have New Relic monitoring those stats and we see no outliers. – Jeppz May 18 '20 at 13:55
  • Have you tried it without the Monitoring thing, if you can get rid of that stuff easily? – tquadrat May 18 '20 at 14:02
  • @tquadrat Yea we read that java agents usage cant be tracked easily so we removed that startup flag but with no change. – Jeppz May 18 '20 at 14:08
  • Can you provide any additional information regarding the increasing size of the memory consumption? Any relevant reports from JVM tracking utilities? It seems like guessing in the dark without any additional information........ – Rann Lifshitz May 24 '20 at 10:36
  • @RannLifshitz sorry but we have yet to find any information on this memory, all we see is increasing process and swap but none of the java tools report on this memory use. – Jeppz May 25 '20 at 07:55
  • @Jeppz Have you tried contacting anyone from Oracle regarding this issue? Via emails/forums/etc... ? They would probably have a better idea of where to look for potential build related memory leaks.... – Rann Lifshitz May 25 '20 at 08:51

2 Answers2

1

In Java 11 the ForkJoinPool class has a slightly different behaviour.

The default elapsed time since last use before a thread is terminated is 60 seconds. In Java 8 this was undocumented but actually hardcoded with 2 seconds. In case of oversized pools, the Java 8 implementation terminates idle threads two seconds after the pool was created. But the Java 9/11 version of the class keeps them alive for minutes.

Compare the number and lifetime of threads. Since unused threads may not be terminated early anymore when your application is started or ForkJoinPools objects are created the extended lifetime of threads could easily lead to a memory issue.

See the following question for a similar problem: ForkJoinPool performance Java 8 vs 11

In Java 9 a new constructor was introduced to configure the value. To get the same behavior as with Java 8 compilation, you have to set keepAliveTime explicitly to 2 seconds or reduce the size of your ForkJoinPool objects before you compile to Java 9.

rmunge
  • 3,653
  • 5
  • 19
  • 1
    Why should that depend on whether the code was compiled with Java 8 or Java 11? – Johannes Kuhn May 20 '20 at 20:48
  • Sorry, I totally missed that also the Java 8 build was executed on a server with Java 11 and not on Java 8. So the changed behavior in ForkJoinPool cannot explain why the Java 8 build works fine on Java 11. – rmunge May 20 '20 at 23:03
  • The behavior of ForkJoinPool has changed, no matter what Java build was used. Therefore I would still suggest to have a closer look on the number of threads. They could be an explanation for the strange memory leak. Maybe there are also other differences between the two builds (e.g. due to multi-release JARs in 3rd party libraries) and maybe the combination makes the difference. – rmunge May 20 '20 at 23:09
  • Hmm interesting point but our New Relic stats do not show an increased amount of threads since before the upgrade and we arent using ForkJoinPool (but I guess a dependency might). – Jeppz May 21 '20 at 21:16
1

After digging a bit into Java's compiler code I found an interesting change, introduced in Java 9 which I wasn't aware of yet. This change CAN cause different behaviour depending on your compilation target:

While its widely known that most optimization are done by the JIT compiler instead of javac, the later still does some code optimizations. Before Java 9, one of these optimizations was the translation of String concatenations into StringBuilder::append chains. Starting with Java 9, javac makes use of invokedynamic calls to the newly introduced java.lang.invoke.StringConcatFactory class instead of translating to StringBuilder:append calls. So when you compile to Java 8, javac will produce optimized byte code, while when you compile to Java 9 optimization is delegated to the mentioned built-in class at runtime.

The corresponding JEP 280 provides some more details about this change. One success metric of the JEP 280 was that String concatenation performance must not regress. But JDK-8221760 already reports a potential performance regression. According to the bug entry, code with string concatenation compiled to Java 8 seems to perform bettern on Java 11u than the same code compiled to Java 9 or 11. The bug entry is still unresolved, so maybe performance is not the only regression here.

rmunge
  • 3,653
  • 5
  • 19
  • But how does that cause a memory leak? Especially an off-heap memory leak? Unless StringConcatFactory leaks memory off heap – Thayne Dec 12 '22 at 23:57