Use LambdaMetafactory to invoke one-arg method on class instance obtained from other classloader

Question

Based on this stackoverflow answer, I am attempting to instantiate a class using reflection and then invoke a one-argument method on it using LambdaMetafactory::metafactory (I tried using reflection, but it was rather slow).

More concretely, I want to create an instance of com.google.googlejavaformat.java.Formatter, and invoke its formatSource() method with the following signature: String formatSource(String input) throws FormatterException.

I have defined the following functional interface:

@FunctionalInterface
public interface FormatInvoker {
  String invoke(String text) throws FormatterException;
}

and am attempting to execute the following code:

try (URLClassLoader cl = new URLClassLoader(urls.toArray(new URL[urls.size()]))) {
  Thread.currentThread().setContextClassLoader(cl);

  Class<?> formatterClass =
      cl.loadClass("com.google.googlejavaformat.java.Formatter");
  Object formatInstance = formatterClass.getConstructor().newInstance();

  Method method = formatterClass.getMethod("formatSource", String.class);
  MethodHandles.Lookup lookup = MethodHandles.lookup();
  MethodHandle methodHandle = lookup.unreflect(method);
  MethodType type = methodHandle.type();
  MethodType factoryType =
      MethodType.methodType(FormatInvoker.class, type.parameterType(0));
  type = type.dropParameterTypes(0, 1);

  FormatInvoker formatInvoker = (FormatInvoker)
    LambdaMetafactory
        .metafactory(
            lookup,
            "invoke",
            factoryType,
            type,
            methodHandle,
            type)
        .getTarget()
        .invoke(formatInstance);

  String text = (String) formatInvoker.invoke(sourceText);
} finally {
  Thread.currentThread().setContextClassLoader(originalClassloader);
}

When I run this code, the call to LambdaMetafactory::metafactory fails with the following exception:

    Caused by: java.lang.invoke.LambdaConversionException: Exception finding constructor
        at java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:229)
        at java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:304)
        at com.mycompany.gradle.javaformat.tasks.JavaFormatter.formatSource(JavaFormatter.java:153)
        ... 51 more
    Caused by: java.lang.IllegalAccessException: no such method: com.delphix.gradle.javaformat.tasks.JavaFormatter$$Lambda$20/21898248.get$Lambda(Formatter)FormatInvoker/invokeStatic
        at java.lang.invoke.MemberName.makeAccessException(MemberName.java:867)
        at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1003)
        at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:1386)
        at java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:780)
        at java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:226)
        ... 53 more
    Caused by: java.lang.LinkageError: bad method type alias: (Formatter)FormatInvoker not visible from class com.delphix.gradle.javaformat.tasks.JavaFormatter$$Lambda$20/21898248
        at java.lang.invoke.MemberName.checkForTypeAlias(MemberName.java:793)
        at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:976)
        at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1000)
        ... 56 more

I've read through a number of stackoverflow answers about LambdaMetafactory and read the LambdaMetafactory documentation, but have not been able to figure out what I am doing wrong. I am hoping that somebody else will be able to.

Thank you in advance for your help.

_"I tried using reflection, but it was rather slow"_. If you plan on creating a new metafactory for each invocation, that would be even slower. If the target method is statically know, why can't you just invoke the method directly? Any ways, I can't reproduce your error, but it looks like the `FormatInvoker` is defined in another package than your snippet's code and it's not accessible? — Jorn Vernee, Jun 10 '18 at 19:24
Actually... I think the problem is that the class hosting the snippet is in a different class loader, since you change the context classloader `Formatter` gets loaded by the new classloader. You could try calling `lookup.in(Formatter.class)` _maybe_ that will work. But I might be easier to just cast the result of `newInstance()` to `Formatter` and then call `formatSource` directly on the result. — Jorn Vernee, Jun 10 '18 at 19:46
@JornVernee thank you for your response! To respond to your suggestions: 1) `FormatInvoker` is actually defined in the same class as the rest of the code in the snippet. 2) Calling `lookup.in(Formatter.class)` didn't seem to do the trick, unfortunately. 3) I am not able to cast the result of `newInstance()` to `Formatter`, as `newInstance()` yields an instance from the new classloader, while `Formatter` is present in the original classloader. It is not possible to cast across classloaders. — Alex Kleiman, Jun 10 '18 at 23:15
OK. Which line actually throws the error? You could also try simplifying, since you don't need the metafactory, you can replace the call to `getMethod` with `lookup().findVirtual(formatterClass, "formatSource", methodType(String.class, String.class))` and then call `invokeExact` on the returned method handle. — Jorn Vernee, Jun 10 '18 at 23:23
It is the actual call to `LambdaMetafactory.metafactory()` which throws the exception. And unfortunately, using `MethodHandle::invokeExact` still doesn't provide me with the performance I am looking for. I am hoping that using a `LambdaMetafactory` will be more performant. — Alex Kleiman, Jun 11 '18 at 01:40
The metafactory just lets you put a type around the methodhandle, it doesn't do anything to be more performant. The idea behind method handles is that you do the access checking once, at lookup, and then cache it for every invocation you need to do. While reflection does the access check on every invocation. Since the method you're trying to invoke is statically known, you can stick the method handle in a `static final` field and initialize it in the class's `static` initializer. The JIT should then also see it as constant and inline it into the call site. — Jorn Vernee, Jun 11 '18 at 07:51

Holger · Accepted Answer · 2018-09-18T08:31:34.933

The MethodHandles.Lookup instance returned by MethodHandles.lookup() encapsulates the caller’s context, that is, the context of your class which creates the new class loader. As the exception says, the type Formatter is not visible from this context. You can see this as an attempt to mimic the compile-time semantics of the operation; if you placed the statement Formatter.formatSource(sourceText) in your code, it wouldn’t work as well, due to the fact that the type is not in scope.

You can change the context class of the lookup object using in(Class), but when using MethodHandles.lookup().in(formatterClass), you’ll run into a different problem. Changing the context class of a lookup object will reduce the access level to align it with the Java access rules, i.e. you can only access public members of the class Formatter. But the LambdaMetafactory only accepts lookup objects having private access to their lookup class, i.e. lookup objects directly produced by the caller itself. The only exception would be changing between nested classes.

Therefore using MethodHandles.lookup().in(formatterClass) results in Invalid caller: com.google.googlejavaformat.java.Formatter, as you (the caller) are not that Formatter class. Or technically, the lookup object has not the private access mode.

The Java API doesn’t offer any (simple) way to get a lookup object to be in a different class loading context and having the private access (prior to Java 9). All regular mechanisms would involve the cooperation of the code residing in that context. That’s the point where developers often go the route of doing Reflection with access override to manipulate the lookup object, to have the desired properties. Unfortunately, the new module system is expected to become more restrictive in the future, likely breaking these solutions.

Java 9 offers a way to get such a lookup object, privateLookupIn, which requires the target class to be in the same module or its module to be opened to the caller’s module to permit such an access.

Since you are creating a new ClassLoader, you have hands on the class loading context. So, one way to solve the problem, is to add another class to it, which creates the lookup object and allows your calling code to retrieve it:

    try (URLClassLoader cl = new URLClassLoader(urls.toArray(new URL[0])) {
        { byte[] code = gimmeLookupClassDef();
          defineClass("GimmeLookup", code, 0, code.length); }             }) {

        MethodHandles.Lookup lookup = (MethodHandles.Lookup)
            cl.loadClass("GimmeLookup").getField("lookup").get(null);
        Class<?> formatterClass =
            cl.loadClass("com.google.googlejavaformat.java.Formatter");

        Object formatInstance = formatterClass.getConstructor().newInstance();

        Method method = formatterClass.getMethod("formatSource", String.class);
        MethodHandle methodHandle = lookup.unreflect(method);
        MethodType type = methodHandle.type();
        MethodType factoryType =
            MethodType.methodType(FormatInvoker.class, type.parameterType(0));
        type = type.dropParameterTypes(0, 1);

        FormatInvoker formatInvoker = (FormatInvoker)
          LambdaMetafactory.metafactory(
                lookup, "invoke", factoryType, type, methodHandle, type)
            .getTarget().invoke(formatInstance);

      String text = (String) formatInvoker.invoke(sourceText);
      System.out.println(text);
    }

static byte[] gimmeLookupClassDef() {
    return ( "\u00CA\u00FE\u00BA\u00BE\0\0\0001\0\21\1\0\13GimmeLookup\7\0\1\1\0\20"
    +"java/lang/Object\7\0\3\1\0\10<clinit>\1\0\3()V\1\0\4Code\1\0\6lookup\1\0'Ljav"
    +"a/lang/invoke/MethodHandles$Lookup;\14\0\10\0\11\11\0\2\0\12\1\0)()Ljava/lang"
    +"/invoke/MethodHandles$Lookup;\1\0\36java/lang/invoke/MethodHandles\7\0\15\14\0"
    +"\10\0\14\12\0\16\0\17\26\1\0\2\0\4\0\0\0\1\20\31\0\10\0\11\0\0\0\1\20\11\0\5\0"
    +"\6\0\1\0\7\0\0\0\23\0\3\0\3\0\0\0\7\u00B8\0\20\u00B3\0\13\u00B1\0\0\0\0\0\0" )
    .getBytes(StandardCharsets.ISO_8859_1);
}

This subclasses URLClassLoader to call defineClass once in the constructor to add a class being equivalent to

public interface GimmeLookup {
    MethodHandles.Lookup lookup = MethodHandles.lookup();
}

Then, the code reads the lookup field via Reflection. The lookup object encapsulates the context of GimmeLookup, which is defined within the new URLClassLoader, and is sufficient to access the public method formatSource of the public com.google.googlejavaformat.java.Formatter.

The interface FormatInvoker will be accessible for that context, as your code’s class loader will become the parent of the created URLClassLoader.

Some additional notes:

Of course, this can only become more efficient than any other reflective access, if you use the generated FormatInvoker instance sufficiently often to compensate for the costs of creating it.
I removed the Thread.currentThread().setContextClassLoader(cl); statement, as it has no meaning in this operation, but is, in fact, quiet dangerous as you didn’t set it back, so the thread kept a reference to the closed URLClassLoader afterwards.
I simplified the toArray call to urls.toArray(new URL[0]). This article provides a really interesting view on the usefulness of specifying the collection’s size to the array.

thank you very much for the fantastic answer. I understand what's going on a lot more now thanks to your thorough and generous explanation! — Alex Kleiman, Jun 12 '18 at 01:09
Unfortunately, the code still seems to be failing on the call to `LambdaMetafactory::metafactory`. Now, the error is `java.lang.NoClassDefFoundError: com/delphix/gradle/javaformat/tasks/JavaFormatter$FormatInvoker`. Is this because `FormatInvoker` is not available on the classpath for some reason? Might it be because the new classpath I'm constructing doesn't have `JavaFormatter` on the classpath? I know that you said it should, as my "code's class loader will become the parent of the created `URLClassLoader`". Do you see anything which I might be doing wrong? — Alex Kleiman, Jun 12 '18 at 01:11
There's nothing directly jumping into my eye. You could try to run it on Java 9 or newer, as these versions provide more detail information with `NoClassDefFoundError`, e.g. if it has been caused by an unresolved indirect dependency. The other option would by to step through the class loading process in a debugger. — Holger, Jun 12 '18 at 06:24
It turns out that I needed to call the constructor `new URLClassLoader(urls.toArray(new URL[0]), Thread.currentThread().getContextClassLoader())` in order for the original classloader to be the parent of the new classloader I am creating here. With that, this solution works! Thank you very much for all your help! — Alex Kleiman, Jun 13 '18 at 04:17

Use LambdaMetafactory to invoke one-arg method on class instance obtained from other classloader

1 Answers1

Linked