After digging a bit into Java's compiler code I found an interesting change, introduced in Java 9 which I wasn't aware of yet. This change CAN cause different behaviour depending on your compilation target:
While its widely known that most optimization are done by the JIT compiler instead of javac, the later still does some code optimizations. Before Java 9, one of these optimizations was the translation of String concatenations into StringBuilder::append chains. Starting with Java 9, javac makes use of invokedynamic calls to the newly introduced java.lang.invoke.StringConcatFactory class instead of translating to StringBuilder:append calls. So when you compile to Java 8, javac will produce optimized byte code, while when you compile to Java 9 optimization is delegated to the mentioned built-in class at runtime.
The corresponding JEP 280 provides some more details about this change. One success metric of the JEP 280 was that String concatenation performance must not regress. But JDK-8221760 already reports a potential performance regression. According to the bug entry, code with string concatenation compiled to Java 8 seems to perform bettern on Java 11u than the same code compiled to Java 9 or 11. The bug entry is still unresolved, so maybe performance is not the only regression here.