How to pass HiveContext as an argument from one function to another function using spark scala

Question

I have a sceneria where I need to pass HiveContext as an argument to another function. Below is my code and where I am stuck with issue:

Object Sample {
    def main(args:Array[String]){
        val fileName = "SampleFile.txt"
        val conf = new SparkConf().setMaster("local").setAppName("LoadToHivePart")  
        conf.set("spark.ui.port","4041")
        val sc=new SparkContext(conf)
        val sqlContext = new org.apache.spark.sql.SQLContext(sc)  
        val hc = new org.apache.spark.sql.hive.HiveContext(sc) 
        hc.setConf("hive.metastore.uris","thrift://127.0.0.1:9083")
        test(hc,fileName)
        sc.stop()
    }
    def test(hc:String, fileName: String){
        //code.....
    }
}

As per above code I am unable to pass a HiveContext variable "hc" from main to another function. Also tried with:

def test(hc:HiveContext, fileName:String){} but it is showing error for both.

If you're using Spark 2, those shouldn't be used anyway – OneCricketeer Jan 06 '18 at 15:38 — OneCricketeer, Jan 06 '18 at 15:38

score 1 · Answer 1 · answered Jan 06 '18 at 10:00

1

def test(hc:HiveContext, fileName: String){
        //code.....
    }

Note: Hive Context available in org.apache.spark.sql.hive.HiveContext

so import it using import org.apache.spark.sql.hive.HiveContext

answered Jan 06 '18 at 10:00

undefined_variable

6,180
2
22
37

@RameshMaharjan import might be missing otherwise it has to be some other error.. – undefined_variable Jan 06 '18 at 10:07
yeah I guess you are right as he initialized using `val hc = new org.apache.spark.sql.hive.HiveContext(sc)`. :) +1 – Ramesh Maharjan Jan 06 '18 at 10:08

How to pass HiveContext as an argument from one function to another function using spark scala

1 Answers1