1

Currently working on converting below code as "JAR" to register permanent UDF in Databricks cluster. facing issue like NoClassDefFoundError, But i added required Library dependencies while building Jar using SBT. source code : https://databricks.com/notebooks/enforcing-column-level-encryption.html

Used Below in build.sbt

scalaVersion := "2.13.4"

libraryDependencies += "org.apache.hive" % "hive-exec" % "0.13.1"

libraryDependencies += "com.macasaet.fernet" % "fernet-java8" % "1.5.0"

Guide me on right libraries if anything wrong above.

Kindly help me on this,

import com.macasaet.fernet.{Key, StringValidator, Token}
import org.apache.hadoop.hive.ql.exec.UDF;

class Validator extends StringValidator {

  override def getTimeToLive() : java.time.temporal.TemporalAmount = {
    Duration.ofSeconds(Instant.MAX.getEpochSecond());
  }
}

class udfDecrypt extends UDF {

  def evaluate(inputVal: String, sparkKey : String): String = {

    if( inputVal != null && inputVal!="" ) {
      val keys: Key = new Key(sparkKey)
      val token = Token.fromString(inputVal)
      val validator = new Validator() {}
      val payload = token.validateAndDecrypt(keys, validator)
      payload
    } else return inputVal
  }
}
Maha
  • 21
  • 2
  • You need either do `sbt assembly` to include this dependency into jar, or add this dependency explicitly to the cluster definition – Alex Ott Dec 23 '20 at 10:12
  • Thanks a lot @AlexOtt! fixed it!. While reading Secret from Spark config getting below error, any thoughts? HiveException: Unable to execute method public java.lang.String udfDecrypt.evaluate(java.lang.String,java.lang.String) with arguments {***gAAAAABf4kVBzP-LI0geSu0hlXq89XwAgxeJ4QZAbYhnj5MobaO9KZxtxsP8XRZ1O6LskZwdI8hr5QH9gtGXYffSaKDwQjLNiQ==,[REDACTED]}:Illegal base64 character 2a Caused by: InvocationTargetException: Caused by: IllegalArgumentException: Illegal base64 character 2a – Maha Dec 23 '20 at 13:40
  • check that your string is valid base64 – Alex Ott Dec 23 '20 at 14:25
  • Sure, Thanks!, let me check! – Maha Dec 23 '20 at 16:16

2 Answers2

0

Make sure the fernet-java library is installed in your cluster. enter image description here

Lucas Sousa
  • 401
  • 5
  • 18
0

This topic is related to

Databricks SCALA UDF cannot load class when registering function

I tried more to install the jar file to the cluster via the Libraries in the config, not drop directly to DBFS as the userguide, then I faced the issue with validator not found and the question routed me here.

I added the maven repo to the Libraries config, but then the cluster failed to installed it, with error

Library resolution failed because unresolved dependency: com.macasaet.fernet:fernet-java8:1.5.0: not found

databricks cluster libraries

Have you experienced with this?

Minh-Dung Dang
  • 1
  • 1