2

I'm using spark 2.2.0, and I would like to create unit tests for spark with hive support. The test should relay on a metastore that is stored on the local disk (as explained in the programming guide)

I define the session in the following way:

val spark = SparkSession
 .builder
 .config(conf)
 .enableHiveSupport()
 .getOrCreate()

the creation of the spark session fails with the error:

org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "datanucleus" yet this has not been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.

I managed to work around this error by adding the following dependency:

"org.datanucleus" % "datanucleus-accessplatform-jdo-rdbms" % "3.2.9"

This is strange to me, since this library is already included in spark.

Is there another way to solve this? I wouldn't wan't to keep track of the library and update it with every new spark version.

lev
  • 3,986
  • 4
  • 33
  • 46

0 Answers0