1

I'm struggling to determine what is causing a moderately large Groovy application to perform slowly in production. When taking thread dumps of running applications the strange thing I'm seeing is lots of threads with a stacktrace like this: 

    at java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(Native Method)
    at java.lang.invoke.CallSite.setTargetNormal(CallSite.java:258)
    at java.lang.invoke.MutableCallSite.setTarget(MutableCallSite.java:154)
    at org.codehaus.groovy.vmplugin.v7.Selector$MethodSelector.doCallSiteTargetSet(Selector.java:909)
    at org.codehaus.groovy.vmplugin.v7.Selector$MethodSelector.setCallSiteTarget(Selector.java:969)
    at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:228)
    at java.lang.invoke.LambdaForm$DMH/1665404403.invokeStatic_L3IL5_L(LambdaForm$DMH)
    at java.lang.invoke.LambdaForm$BMH/1705072168.reinvoke(LambdaForm$BMH)

I've seen some mentions online that the setCallSiteTargetNormal method is rather heavyweight and blocks all JVM threads while it is invoked. We use Groovy's invoke dynamic support and I'm wondering if we are hitting some kind of bug in Groovy which is causing this method to be called too extensively.

Obvious performance issues I've already checked:

  • Memory usage is fine
  • GC overhead is normal
  • CPU usage on the server looks ok
  • External DB and web service calls are all normal

One note about CPU usage is that the app is behaving like it is starved for CPU, but it only is using about about 1/4 of the total CPU of a 4 CPU machine. It is acting like a thread should be pegging a CPU a 100%, but I'm not seeing that at all. However some information I've found online points to the setCallSiTeTargetNormal method call as being something that completely blocks all threads on the JVM.

jjathman
  • 12,536
  • 8
  • 29
  • 33
  • have u hit this? https://bugs.openjdk.java.net/browse/JDK-8151981 – Eugene Aug 28 '17 at 13:32
  • Yeah, I think that's the same. I ended up filing a Groovy bug about it since I'm pretty sure it is a Groovy issue. https://issues.apache.org/jira/browse/GROOVY-8298 – jjathman Aug 29 '17 at 14:59

1 Answers1

0

Let me propose another alternative to the setCallSiTeTargetNormal method. In such a problem, I would check GC logs in order to see if it is entering too often to clean Permgen (for Java 7 and before) or Metaspace (for Java 8 and later). Metaspace is automatically cleaned if you specify a limit for it, while for Permgen you need certain switches in the command line.

When used to execute a script inside a running JVM, Groovy makes intensive use of such space in order to instantiate class loaders for every script. Then, when it reaches its limit, GC must clean it. This is my scenario when using Jenkins, for example. Thus, using Metaspace as if it was some kind of heap, with the same behavior. Every GC sweep should stop your app, causing noticeable stops, while not hitting CPU, etc.

You should be able to test and maybe discard this idea by using this command line to start your application and see how often the GC makes the complete sweep (and how long is the stop).

-XX:+PrintGC -XX:+PrintGCTimeStamps -Xloggc:/my/log/file

Jorge_B
  • 9,712
  • 2
  • 17
  • 22