0

Followed the steps mentioned in this docs. databricks-connect test command works fine. However, when I launch the test scala program from Intellij, I'm seeing following error:

Exception in thread "main" java.lang.NoSuchMethodError: com.databricks.spark.util.MetricDefinitions$.EVENT_INITIAL_CONFIG_LOG()Lcom/databricks/spark/util/MetricDefinition;
    at com.databricks.spark.util.InitialConfigLogging$.recordInitialConfigs(InitialConfigLogging.scala:47)
    at org.apache.spark.sql.internal.SQLConf.recordSessionInitialConfs(SQLConf.scala:4166)
    at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:150)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1033)
    at com.srikanth.Demo$.main(Demo.scala:13)
    at com.srikanth.Demo.main(Demo.scala)

Environment details:

python - 3.8
java - 1.8
databricks-connect - 9.1
TeluguCoder
  • 1
  • 2

1 Answers1

0

I had the same error message, turns out I did not add the databricks-connect library properly. You should add the library like described here: https://docs.databricks.com/dev-tools/databricks-connect.html#intellij-scala-or-java

fv_3
  • 31
  • 3