Scala Class Loader for Avro Tools Run Method Written in Java

Question

I'm having some difficulty in determining the means for loading an "Avro Tools" class and its run method. The issue is somewhere between java and scala interfacing and class loading methods. Due to the fact that avro is used elsewhere in a Spark app with a different version for loading data files, I need to be able to treat this particular method as a siloed call to another version of avro-tools.

The following is my code:

package samples

import java.io.{ByteArrayOutputStream, InputStream}
import org.junit.runner.RunWith
import org.specs2.mutable._
import org.specs2.runner._

import scala.collection.JavaConverters._

@RunWith(classOf[JUnitRunner])
class MySpecTest extends Specification {
  "Class Loader" should {
    "load an implement a class" in {

      var classLoader = new java.net.URLClassLoader(
        Array(new java.io.File("./avro-tools-1.9.1.jar").toURI.toURL),
        this.getClass.getClassLoader)

      var clazzDFRT = classLoader.loadClass("org.apache.avro.tool.DataFileRepairTool")

      val objDFRT = clazzDFRT.getConstructor().newInstance()
      val toolCmdArgsAsJava = List("-o", "all", "questionable.avro", "fixed.avro").asJava
      val stdin : InputStream = null
      val out: ByteArrayOutputStream = new ByteArrayOutputStream
      val stdout = new PrintStream(out) // added stdout in edit#1

      val err = System.err
      val toolClassArgsAsJava = List(stdin, stdout, // changed out to stdout in edit#1 
          err, toolCmdArgsAsJava).asJava 

      //  parameterTypes: Class[_] *
      //  public int run( InputStream stdin, PrintStream out, PrintStream err, List<String> args)

      val paramClasses: Array[Class[_]] = Array(classOf[InputStream], classOf[PrintStream], classOf[PrintStream], classOf[java.util.List[_]])

      val method = clazzDFRT.getMethod("run",  paramClasses : _*)

      // the following produces wrong number of arguments exception
      method.invoke(objDFRT.asInstanceOf[Object], toolClassArgsAsJava)

      // sidebar: is this the end result for the Unit test - want out str with summary
      out.toString("UTF-8").contains("File Summary")
    }
  }
}

I seem to have some issue in the invoke method part, but maybe the whole solution is a little off - I need to be able to invoke the method as well as load, instantiate or ...

How can I fix this to run the entire code segment (and repair a broken avro)?

iain · Answer 1 · 2019-10-01T08:30:22.267

It is hard to tell the exact nature of the problem since you didn't include the exception or stack trace. I am not sure why you are loading the avro tools dynamically instead of including the jar statically as part of your build.

//  public int run( InputStream stdin, PrintStream out, PrintStream err, List<String> args)
val method = clazzDFRT.getMethod("run",  Class[_] : _*)

You are not specifying the parameters correctly.

method.invoke(objDFRT.asInstanceOf[Object], toolClassArgsAsJava)

val params: Array[Class[_]] = Array(classOf[InputStream], classOf[PrintStream], classOf[PrintStream], classOf[java.util.List[_]])
val method = clazzDFRT.getMethod("run",  params : _*)

or

val method = clazzDFRT.getMethod("run", classOf[InputStream], classOf[PrintStream], classOf[PrintStream], classOf[java.util.List[_]])

To fix the invoke, you cannot pass the parameters in a list. The invoke method takes variable arguments, you need to pass these in directly.

method.invoke(objDFRT.asInstanceOf[Object], stdin, stdout, stderr, toolCmdArgsAsJava)

or

method.invoke(objDFRT.asInstanceOf[Object], Array(stdin, stdout, stderr, toolCmdArgsAsJava): _*)

Notice the second option uses an Array not a List.

I suggest you read up on the documentation for using var args in Java and Scala * https://docs.oracle.com/javase/8/docs/technotes/guides/language/varargs.html * http://daily-scala.blogspot.com/2009/11/varargs.html

Thx. That helps me get past the part about loading the method. I'm stuck on the invoking part now. Sidebar: I'm loading the jar as another code part uses the older version of the avro libraries and I need the tool from the newer version. — codeaperature, Sep 30 '19 at 18:36

Scala Class Loader for Avro Tools Run Method Written in Java

1 Answers1