0

I am trying to access my existing hadoop setup in my spark+scala project

Spark Version 1.4.1 Hadoop 2.6 Hive 1.2.1

from Hive Console I able to create table and access it without any issue, I can also see the same table from Hadoop URL as well.

the problem is when I try to create a table from project, system shows error

ERROR Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:file:/user/hive/warehouse/src is not a directory or unable to create one)

following is the code I write:

import

import org.apache.spark._
import org.apache.spark.sql.hive._

Code

val sparkContext = new SparkContext("local[2]", "HiveTable")
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sparkContext)
hiveContext.setConf("hive.metastore.warehouse.dir", "hdfs://localhost:54310/user/hive/warehouse")  
hiveContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")

Edit: instead of create table if I had to execute insert statement like:

hiveContext.sql("INSERT INTO TABLE default.src SELECT 'username','password' FROM foo;")

any help to resolve his issue would be highly appreciable.

Nihal Bhagchandani
  • 239
  • 5
  • 9
  • possible duplicate of http://stackoverflow.com/questions/25989187/unable-to-create-table-in-hive-reading-a-csv-from-hdfs – Bacon Aug 06 '15 at 02:19
  • @Bacon I have issue when setting up a Apache Spark on Hive in scala project, where in the link you provided is not related. – Nihal Bhagchandani Aug 06 '15 at 04:11
  • 1
    you need to have hive-site.xml in your spark classpath.. so that all properties related to hive-site.xml are available to spark. Copy hive-site.xml from your Hive cluster to your $SPARK_HOME/conf/ – vijay kumar Aug 06 '15 at 07:17

0 Answers0