0

I have been connecting to Hive (which is in cluster) from my local system with spark in scala IDE , finally placed my hive-site.xml properly in Spark/conf folder and in class path and able to connect to metastore but unable to access hive tables.

How to change the user for HiveContext

15/12/22 10:28:42 INFO ParseDriver: Parsing command: show tables 15/12/22 10:28:43 INFO ParseDriver: Parse Completed 15/12/22 10:28:45 INFO metastore: Trying to connect to metastore with URI thrift://Server.com:9083 15/12/22 10:28:46 INFO JniBasedUnixGroupsMapping: Error getting groups for XXXX: The user name could not be found.

15/12/22 10:28:46 WARN UserGroupInformation: No groups available for user XXXXX 15/12/22 10:28:46 INFO metastore: Connected to metastore.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:353) at org.apache.spark.sql.hive.HiveContext$$anonfun$4.apply(HiveContext.scala:235) at org.apache.spark.sql.hive.HiveContext$$anonfun$4.apply(HiveContext.scala:231) at scala.Option.orElse(Option.scala:257) at org.apache.spark.sql.hive.HiveContext.x$3$lzycompute(HiveContext.scala:231) at org.apache.spark.sql.hive.HiveContext.x$3(HiveContext.scala:229) at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:229) at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:229) at org.apache.spark.sql.hive.HiveMetastoreCatalog.(HiveMetastoreCatalog.scala:55) at org.apache.spark.sql.hive.HiveContext$$anon$1.(HiveContext.scala:253) at org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:253) at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:253) at org.apache.spark.sql.hive.HiveContext$$anon$3.(HiveContext.scala:263) at org.apache.spark.sql.hive.HiveContext.analyzer$lzycompute(HiveContext.scala:263) at org.apache.spark.sql.hive.HiveContext.analyzer(HiveContext.scala:262) at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:411) at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:411) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.(SchemaRDD.scala:108) at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:102) at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:106) at com.Spark.ConnectToHadoop.CountWords$.main(CountWords.scala:50) at com.Spark.ConnectToHadoop.CountWords.main(CountWords.scala) Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.api.SessionNotRunning at java.net.URLClassLoader$1.run(Unknown Source) at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 23 more

sudhir
  • 1,387
  • 3
  • 25
  • 43
  • Looks like your Hadoop cluster does not use Kerberos authentication, so the answer is somewhere in that post >> http://stackoverflow.com/questions/11041253/set-hadoop-system-user-for-client-embedded-in-java-webapp/11062529#11062529 – Samson Scharfrichter Dec 22 '15 at 07:39
  • Anyway that will make **no difference** because your Exception has nothing to do with authentication *(which just displays a Warning)*; your `hive-site.xml` clearly invokes TEZ but you did not provide TEZ libraries to Spark Class Path. – Samson Scharfrichter Dec 22 '15 at 07:42
  • @SamsonScharfrichter Its not just warning , I'm unable to load my hive tables because of the exception , anyway how can I add Tez Libraries (where should I download it) and does that clear my exception ? – sudhir Dec 22 '15 at 08:33
  • **1.** Spark SQL is supposed to make TEZ *irrelevant*, so just use my answer below and do not use TEZ -- until you understand what you are doing **2.** Again, the exception has no relation to the warning above!!! – Samson Scharfrichter Dec 22 '15 at 09:07

1 Answers1

1

This answer that has nothing to do with your question, but then, your question has nothing to do with the Exception shown...

In the hive-site.xml that you provided to Spark, change propertyhive.execution.engine from "tez" (requires TEZ run-time) to default "mr".

Samson Scharfrichter
  • 8,884
  • 1
  • 17
  • 36
  • Thanks, that really helped.. But how to run with tez engine , will spark only work with 'mr' on Hive?? – sudhir Dec 22 '15 at 11:00