2

With Spark 1.5.1, I've already been able to access spark-shell temporary tables from Beeline using Thrift Server. I've been able to do so by reading answers to related questions on Stackoverflow.

However, after upgrading to Spark 2.0, I can't see temporary tables from Beeline anymore, here are the steps I'm following.

I'm launching spark-shell using the following command:

./bin/spark-shell --master=myHost.local:7077 —conf spark.sql.hive.thriftServer.singleSession=true

Once the spark shell is ready I enter the following lines to launch thrift server and create a temporary view from a data frame taking its source in a json file

import org.apache.spark.sql.hive.thriftserver._

spark.sqlContext.setConf("hive.server2.thrift.port","10002")
HiveThriftServer2.startWithContext(spark.sqlContext)
val df = spark.read.json("examples/src/main/resources/people.json")
df.createOrReplaceTempView("people")
spark.sql("select * from people").show()

The last statement displays the table, it runs fine.

However when I start beeline and log to my thrift server instance, I can't see any temporary tables:

show tables;
+------------+--------------+--+
| tableName  | isTemporary  |
+------------+--------------+--+
+------------+--------------+--+
No rows selected (0,658 seconds)

Did I miss something regarding my spark upgrade from 1.5.1 to 2.0, how can I gain access to my temporary tables ?

Cedric Artigue
  • 21
  • 3

1 Answers1

0

This worked for me after upgrading to spark 2.0.1

 val sparkConf = 
        new SparkConf()
            .setAppName("Spark Thrift Server Demo")
            .setMaster(sparkMaster)
            .set("hive.metastore.warehouse.dir", hdfsDataUri + "/hive")

      val spark = SparkSession
      .builder()
      .enableHiveSupport()
      .config(sparkConf)
      .getOrCreate()

  val sqlContext = new org.apache.spark.sql.SQLContext(spark.sparkContext) 
      HiveThriftServer2.startWithContext(sqlContext)
user2608613
  • 352
  • 2
  • 11