3

I'm writing an HTML templating language in Kotlin.

My templating engine will need to resolve property expressions, such as obj.myProperty, by looking up "myProperty" not just among the members defined in obj's class and superclasses, but also among extension properties defined in a list of user-specified Kotlin packages.

For instance, if my interpreter is evaluating x.absoluteValue and x turns out to be an Int, I have the following pieces of information:

  • object KClass: Int::class
  • property name: "absoluteValue"
  • list of packages the user asked to search over: kotlin.math , etc.

What API can I use to get a list of all the top-level extension properties defined in a given package, say kotlin.math, as a list of reflected items, such as a List<KProperty<*>>? At template compile time (which is Kotlin runtime) I will go through that list of extensions and look for one named "absoluteValue" compatible with an Int receiver.

I know I can manually define a list of extension properties, such as listOf(Int::absoluteValue, ...) after importing them, but I would like my users to specify a list of packages, not single properties.


Update: I decided to base my template engine on Kotlin's JSR-223 support, with javax.script.ScriptEngineManager, therefore using a stable API and letting the Kotlin compiler resolve extension properties as it sees fit.

Tobia
  • 17,856
  • 6
  • 74
  • 93
  • Wouldn't [kotlin.reflect](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.reflect.full/member-extension-functions.html) work? If you know the package, preferably the class, where the extension is then you can loop through classes in package and check if it contains extension function via name. – chris Aug 27 '19 at 19:09
  • Also, you can use a classpath scanner like [ClassGraph](https://github.com/classgraph/classgraph) to get the classes in a package ([example](https://gist.github.com/ushort/cf017d4c2038d7d53eb8328c9e10698f)). Sadly, as of now I believe its not possible find top-level extensions using reflection so you're stuck with `::`. – chris Aug 27 '19 at 19:31
  • As a clarifying question: Are you looking for something that would work on the jvm only, or are you looking for kotlin-native/kotlin-js solutions as well? – PiRocks Aug 29 '19 at 08:02
  • If you're looking for a jvm only solution you could use standard java bytecode parsing. It gets rather tricky b/c you need to figure what is a extension function from the classfile only. This is on my todo list since its an interesting problem, but may take a few weeks. – PiRocks Aug 29 '19 at 09:30
  • @chris I looked at kotlin.reflect, but didn't find a pointer from a package to a list of its classes and functions. If you have a solution, please post it. – Tobia Aug 29 '19 at 17:33
  • @PiRocks JVM-only is OK, although cross-platform would be preferable. I have no idea what you mean by "standard java bytecode parsing." If you can code a function that goes from [`"kotlin.math"`](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.math/index.html) to some `listOf(Double::absoluteValue, Float::absoluteValue, Int::absoluteValue, Long::absoluteValue, Double::sign, Float::sign, Int::sign, Long::sign, ...)` please post it as an answer! I tried various things but couldn't get anything to work. – Tobia Aug 29 '19 at 17:39
  • @Tobia Sorry, I meant to say to use kotlin.reflect for access to extensions . Afaik Java and Kotlin do not have a way to access classes via package. Alternatives are to use [Reflections](https://github.com/ronmamo/reflections), [ClassGraph](https://github.com/classgraph/classgraph), or [Guava](https://github.com/google/guava). Any of them should be able to give you a list of classes belonging to a package and from there you iterate through the classes, access it's KClass and retrieve extensions like I did in my example code in previous post. – chris Aug 30 '19 at 08:17
  • However, my previous example won't work for top level extensions. For top level extensions, instead of grabbing the `KClass`, you'll use its `Class`, iterate through the methods then get the `kotlinFunction` that represents the java method. Unfortunately, I seem to be unable get the `kotlinFunction` for the `absoluteValue` extension and some others (they return as null) which seems to have to do with them being properties (which should be retrieved using `Field.kotlinProperty`) but being marked as methods in JVM? – chris Aug 30 '19 at 09:11
  • Update: I decided to base my template engine on Kotlin's JSR-223 support (`javax.script.ScriptEngineManager`) so this question is not relevant to me anymore. But the bounty remains, if anybody can solve it. – Tobia Aug 30 '19 at 10:10
  • @Tobia Did my answer cover everything you where looking for? Am I missing something? – PiRocks Sep 01 '19 at 13:42

1 Answers1

3

Something to know about kotlin extension functions:

  • From the point of view of java, kotlin extension functions are static methods which take the class they are extending as a parameter

This makes them hard to distinguish from regular old static functions. Initially I wasn't even sure of there was a difference.

So lets figure out if there is a difference.

Declarations in ExtensionFunctions.kt:

class Test

fun bar(test : Test){}

fun Test.bar2(){}

fun Test.foo45(bar : Test, i :Int): Int = i

Some command line:


francis@debian:~/test76/target/classes/io/github/pirocks$  javap -p -c -s -l ExtensionFunctionsKt.class 
Compiled from "ExtensionFunctions.kt"
public final class io.github.pirocks.ExtensionFunctionsKt {
  public static final void bar(io.github.pirocks.Test);
    descriptor: (Lio/github/pirocks/Test;)V
    Code:
       0: aload_0
       1: ldc           #9                  // String test
       3: invokestatic  #15                 // Method kotlin/jvm/internal/Intrinsics.checkParameterIsNotNull:(Ljava/lang/Object;Ljava/lang/String;)V
       6: return
    LineNumberTable:
      line 6: 6
    LocalVariableTable:
      Start  Length  Slot  Name   Signature
          0       7     0  test   Lio/github/pirocks/Test;

  public static final void bar2(io.github.pirocks.Test);
    descriptor: (Lio/github/pirocks/Test;)V
    Code:
       0: aload_0
       1: ldc           #19                 // String $this$bar2
       3: invokestatic  #15                 // Method kotlin/jvm/internal/Intrinsics.checkParameterIsNotNull:(Ljava/lang/Object;Ljava/lang/String;)V
       6: return
    LineNumberTable:
      line 8: 6
    LocalVariableTable:
      Start  Length  Slot  Name   Signature
          0       7     0 $this$bar2   Lio/github/pirocks/Test;

  public static final int foo45(io.github.pirocks.Test, io.github.pirocks.Test, int);
    descriptor: (Lio/github/pirocks/Test;Lio/github/pirocks/Test;I)I
    Code:
       0: aload_0
       1: ldc           #23                 // String $this$foo45
       3: invokestatic  #15                 // Method kotlin/jvm/internal/Intrinsics.checkParameterIsNotNull:(Ljava/lang/Object;Ljava/lang/String;)V
       6: aload_1
       7: ldc           #24                 // String bar
       9: invokestatic  #15                 // Method kotlin/jvm/internal/Intrinsics.checkParameterIsNotNull:(Ljava/lang/Object;Ljava/lang/String;)V
      12: iload_2
      13: ireturn
    LineNumberTable:
      line 10: 12
    LocalVariableTable:
      Start  Length  Slot  Name   Signature
          0      14     0 $this$foo45   Lio/github/pirocks/Test;
          0      14     1   bar   Lio/github/pirocks/Test;
          0      14     2     i   I

  <further output omitted>
francis@debian:~/test76/target/classes/io/github/pirocks$ 


As you can see there isn't much difference between a regular static function and an extension function except for the first parameter name. Extension function parameters are named $this$functionName. We can use this to figure out whether a function is of the extension varity by parsing the bytecode, and checking parameter names. It is worth mentioning this is somewhat hacky and probably won't work if the classes in question has been run through a bytecode obfuscator. Because writing bytecode parsers on your own is a lot of work I'm using commons-bcel to do all the work for me.

ExtensionFunctions.kt:

package io.github.pirocks
import org.apache.bcel.classfile.ClassParser

class Test

fun bar(test : Test){}

fun Test.bar2(){}

fun Test.foo45(bar : Test, i :Int): Int = i


fun main(args: Array<String>) {
    val classFileInQuestionStream = "Just wanted an object instance".javaClass.getResourceAsStream("/io/github/pirocks/ExtensionFunctionsKt.class")!!
    val parsedClass = ClassParser(classFileInQuestionStream, "ExtensionFunctionsKt.class").parse()
    parsedClass.methods.forEach { method ->
        if(method.localVariableTable.localVariableTable.any {
            it.name == ("\$this$${method.name}")
        }){
            println("Is an extension function:")
            println(method)
        }
    }

}

The above should output:

Is an extension function:
public static final void bar2(io.github.pirocks.Test $this$bar2) [RuntimeInvisibleParameterAnnotations]
Is an extension function:
public static final int foo45(io.github.pirocks.Test $this$foo45, io.github.pirocks.Test bar, int i) [RuntimeInvisibleParameterAnnotations]

Commons-bcel can also provide you with type/name/attribute information for each extension function.

You mentioned in your question doing this with extension functions on Int. This is trickier, because absoluteValue is declared, who knows where(Intellij Ctrl+B tells me that it is located in this massive file called MathH.kt, which is actually MathKt.class, in the package kotlin.math, in some random jar included from maven). Since not everyone will have the same random jar from maven, the best course of action is to look for the kotlin standard library in System.getProperty("java.class.path"). Annoyingly absoluteValue is declared as an inline function and so there is no trace of it in the stdlib jars. This isn't true for all kotlin stdlib extension functions. So you can use the below to get all extension functions in the stdlib (correction: there are two stdlib jars, so this only gets extension functions declared in kotlin-stdlib-version-number).

package io.github.pirocks

import org.apache.bcel.classfile.ClassParser
import java.nio.file.Paths
import java.util.jar.JarFile


class Test

fun bar(test: Test) {}

fun Test.bar2() {}

fun Test.foo45(bar: Test, i: Int): Int = i


fun main(args: Array<String>) {
    val jarPath = System.getProperty("java.class.path").split(":").filter {
        it.contains(Regex("kotlin-stdlib-[0-9]\\.[0-9]+\\.[0-9]+\\.jar"))
    }.map {
        Paths.get(it)
    }.single()//if theres more than one kotlin-stdlib we're in trouble

    val theJar = JarFile(jarPath.toFile())
    val jarEntries = theJar.entries()

    while (jarEntries.hasMoreElements()) {
        val entry = jarEntries.nextElement()
        if (entry.name.endsWith(".class")) {
            val cp = ClassParser(theJar.getInputStream(entry), entry.getName())
            val javaClass = cp.parse()
            javaClass.methods.forEach { method ->
                if (method.localVariableTable?.localVariableTable?.any {
                        it.name == ("\$this$${method.name}")
                    } == true) {
                    println("Is an extension function:")
                    println(method)
                }

            }
        }


    }
}


Edit:

As to actually answering the question about how to get extension functions in a package:

You would need to iterate over each entry in the classpath, both classes and jar, and check for any classes matching the desired package. As for determining the package of a class you can use the commons-bcel function JavaClass::getPackageName.

PiRocks
  • 1,708
  • 2
  • 18
  • 29