2

I am running Spark 2.1 on Mapr 5.0

I am getting following exception while launching Spark on local mode

My spark-default (Important configuration)

spark.sql.hive.metastore.version 0.13.1
spark.sql.hive.metastore.jars 
                          /opt/mapr/lib/maprfs-5.0.0-mapr.jar:
                          /opt/mapr/hadoop/hadoop-0.20.2/conf:
                          /opt/mapr/hadoop/hadoop-0.20.2/lib/protobuf-java-2.5.0.jar:

                          /opt/hadoopgpl/lib/hadoop-lzo.jar:
                          /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar:
                          /opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.1.1.jar:
                          /opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-auth-2.7.0-mapr-1506.jar:
                          /opt/mapr/lib/libprotodefs-5.0.0-mapr.jar:
                          /opt/mapr/lib/baseutils-5.0.0-mapr.jar:
                          /opt/mapr/hadoop/hadoop-0.20.2/lib/guava-13.0.1.jar:
                          /opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-common-2.7.0-mapr-1506.jar:
                          /opt/mapr/hadoop/hadoop-0.20.2/lib/commons-configuration-1.6.jar

spark.sql.hive.metastore.sharedPrefixes com.mysql.jdbc,org.postgresql,com.microsoft.sqlserver,oracle.jdbc,com.mapr.fs.shim.LibraryLoader,com.mapr.security.JNISecurity,com.mapr.fs.jni,com.mapr.fs.shim


java.lang.LinkageError: loader (instance of  org/apache/spark/sql/hive/client/IsolatedClientLoader$$anon$1): attempted  duplicate class definition for name: "com/mapr/fs/jni/MapRConstants"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
... 104 more
java.lang.IllegalArgumentException: Error while instantiating   
'org.apache.spark.sql.hive.HiveSessionState':

Please help me on that

user2553672
  • 101
  • 1
  • 7

1 Answers1

0

I have experienced the same issue. When running my Spark job using spark-submit everything worked but running it locally reproduced the same problem.

Digging out into MapR community yielded this post: https://community.mapr.com/thread/21262-spark-todf-returns-linkage-error-duplicate-class-definition#comments

In addition you will notice that in your cluster in this file spark-defaults.conf there is the following config key: spark.sql.hive.metastore.sharedPrefixes

So, adding this key spark.sql.hive.metastore.sharedPrefixes in SparkConf fixed my problem.

Here is an explanation of this key:

a comma separated list of class prefixes that should be loaded using the classloader that is shared between Spark SQL and a specific version of Hive. An example of classes that should be shared is JDBC drivers that are needed to talk to the metastore. Other classes that need to be shared are those that interact with classes that are already shared. For example, custom appenders that are used by log4j.

You can read more about it in here: https://spark.apache.org/docs/2.1.0/sql-programming-guide.html

boger
  • 553
  • 1
  • 5
  • 12