I have Scala project build with scala verison 2.12 and I can't change scala version of the project. I have create jar file of this project let's call it scalaproject.jar and now I want to include this jar file in spark session. When I create spark session by doing :
spark = SparkSession.builder.config("spark.driver.extraClassPath", "scalaproject.jar").getOrCreate()
I get below error:
Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
at scala.xml.Null$.<init>(Null.scala:23)
at scala.xml.Null$.<clinit>(Null.scala)
at org.apache.spark.ui.jobs.AllJobsPage.<init>(AllJobsPage.scala:43)
at org.apache.spark.ui.jobs.JobsTab.<init>(JobsTab.scala:45)
at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:61)
at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:80)
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:175)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:444)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:834)
From this post, I know that I am getting this issue because of version mismatch. My project is using scala 2.12 and SageMaker notebook is using scala 2.11.
I have scala2.12.jar that I want to send to sparksession so when spark session is create, it can use scala2.12.jar to run scalaproject.jar.
I tried adding scala2.12.jar by doing:
spark = SparkSession.builder.config("spark.driver.extraLibraryPath", "scala2.12.jar").config("spark.driver.extraClassPath", "scalaproject.jar").getOrCreate()
But error is still the same. Can someone please guide me as to how can I add scala2.12.jar in to sparksession in SageMaker notebook?
