2

Background

I am analyzing a performance issue on a Java servlet running on AdoptOpenJDK 11. The response time of the servlet slows down as the number of concurrent requests increases.

To isolate the cause of the issue, I had collected performance information with two tools while changing the number of concurrent requests:

  • Windows Performance Monitor (to inspect CPU usage of the server)
  • JDK Flight Recorder and JDK Mission Control (to inspect synchronization issue in JavaVM)

Problem

According to Mission Control, the servlet has a synchronization issue. It causes so many synchronization events when two or more requests come concurrently. The synchronization events are fired in a class used to lookup a table. The lookup table is implemented with ConcurrentHashMap. So the synchronization is not intentional and looks unnecessary.

On the other hand, according to the Performance Monitor, the number of concurrent requests and the CPU usage % are almost linearly increase together. This is an unexpected result for me.

I had expected that the CPU usage will become constant because the requests will be processed one by one due to the synchronization. As a result of more research, I have found the Java VM had introduced the Adaptive Spinning on Java 6, and I had set a hypothesis that this (spinlock for synchronization in Java VM) is the reason why the CPU usage increased linearly.

Question

The class that causes the synchronization issue is used in so many places in our application. To reasoning and explanation for the change to the class (remove the synchronized block from the class), I have to confirm the hypothesis with the result of a performance test. To confirm the hypothesis, I want to disable the spinlock for synchronization in Java VM.

I have found the JRockit and OpenJ9 had the command-line option to change the behavior of the adaptive spinning. But I could not find the equivalent for OpenJDK. There were no options related to spinlock in the result of java -XX:+PrintFlagsFinal -version.

Is there any way to disable the spinlock for synchronization in OpenJDK Java VM?

SATO Yusuke
  • 1,600
  • 15
  • 39
  • I am looking at the possible flags [here](https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/globals.hpp) and I can't see anything related :( unless Pangin comes with his magic somehow, I do not think this is possible. – Eugene May 05 '21 at 01:29
  • 1
    “…because the requests will be processed one by one due to the synchronization” the frameworks are trying very hard to prevent that. Are you saying that you are trying to sabotage these intentions by performing synchronization on a global object or do you assume that your servlet container has such a synchronization? – Holger May 05 '21 at 07:19
  • 3
    If you want to disable spinning just to check if it makes CPU usage constant - I can tell without any experiment: it won't. To find out, where the CPU time is *really* spent, use [async-profiler](https://github.com/jvm-profiling-tools/async-profiler/). Besides the application code, it shows the most CPU intensive JVM and OS activities, too, including spin locks, etc. Unfortunately, async-profiler does not work on Windows, but you may try running your application on [WSL2](https://learn.microsoft.com/en-us/windows/wsl/) or in a Linux VM. – apangin May 05 '21 at 10:26
  • 1
    @apangin I can confirm that async-profiler does run with WSL2. It can only profile the processes launched within the environment, but for non-UI code that feasible. – Holger May 05 '21 at 12:16
  • @Holger, The synchronization caused in a class in our application, and the class is used in so many places. I am trying to make reasoning and explanation for the change to the class with the result of an experiment that supports the hypothesis. I have added a detailed explanation of the situation in "Problem" and "Question" section. – SATO Yusuke May 05 '21 at 16:14
  • 1
    This doesn’t sound like a synchronization that would cause requests to be processed one by one but a stop-and-go that can perfectly overlap with other executions. It’s not even clear whether the requests are using the same object to synchronize. – Holger May 06 '21 at 06:53

1 Answers1

3

The question is actually an XY problem. I'll answer it with the caveat that the experiment might be useless or even misleading.

The hypothesis that disabled spinning will decrease CPU usage is not necessary correct. Without spinning, a contended lock ends up in calling into OS kernel to park/unpark threads. A system call with a context switch is expensive, so, depending on the number of threads and the length of the critical section, this may actually increase CPU usage.

The options to disable spinning on intrinsic locks in OpenJDK 11 are:

-XX:+UnlockExperimentalVMOptions -XX:SyncKnobs=SpinLimit=0:SpinBase=0:PreSpin=0:SpinEarly=0

However, SyncKnobs have never been found useful, and were completely removed in JDK 12.

apangin
  • 92,924
  • 10
  • 193
  • 247