1

I am working on connecting to data in Hadoop that allows dynamic data type connections.

I need to be able to connect to Hive Thrift Server A, pull in some data, and then connect to Hive Thrift Server B and pull in more data.

To my understanding enableHiveSupport needs to be set up on the initial SparkSession.builder. Is there a way to add/change a thrift connection after the fact?

The only possible solution I have come across is using newSession():SparkSession however I am not sure if this is the correct solution that I am looking for.

I am using Spark 2.1, Hadoop 2.7

Rick Moritz
  • 1,449
  • 12
  • 25
Ryan
  • 86
  • 8
  • You can try to alter `hive.metastore.uris` in the SparkConf at runtime – OneCricketeer Feb 20 '18 at 00:07
  • I tried changing `hive.metastore.uris` during runtime using `sparkSession.conf.set("hive.metastore.uris", "thrift://B:port")` but when I run `sparkSession.sql("SHOW DATABASES").show` I still get back the original database list from thrift A. – Ryan Feb 20 '18 at 13:31
  • Hmm. Well, if your tables aren't in some weird format, you could just read the filesystem location behind both tables – OneCricketeer Feb 20 '18 at 13:51
  • That is a good idea. Unfortunately that is not ideal for what I am working with. – Ryan Feb 20 '18 at 14:36
  • Found a similar question with a solution. https://stackoverflow.com/questions/44949246/can-we-able-to-use-mulitple-sparksessions-to-access-two-different-hive-servers – Ryan Jan 05 '20 at 16:12

2 Answers2

3

According to the book Spark The Definitive Guide "After you create [SparkConf], the SparkConf is immutable for that specific Spark Application" Chambers and Zaharia (2018, p.276)

Ryan
  • 86
  • 8
0
   val spark = SparkSession.builder()
      .appName("Example")
      .master("local")
      .config("hive.metastore.uris","thrift://B:PortNumber")
      .enableHiveSupport()
      .getOrCreate()
Yaswanth Kumar Alla
  • 1
  • 1
  • 4
  • 1
    This builds a new SparkSession, but that doesn't work very well if you already have a SparkSession established. – skylerl Mar 05 '18 at 18:10