1

IN Spark version 1.*

Created emptyRDD like below:

var baseDF = hiveContextVar.createDataFrame(sc.emptyRDD[Row], baseSchema)

While migrating to Spark 2.0(since hiveContext got deprecated, using sparkSession)

Tried like: 

var baseDF = sparkSession.createDataFrame(sc.emptyRDD[Row], baseSchema)

Though getting below error:

org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243)

Is there a way to create emptyRDD using sparkSession?

Shabhana
  • 59
  • 2
  • 2
  • 12
  • 1
    That's not what causing the error. Something else in your code is ! – eliasah Jul 30 '18 at 08:55
  • 2
    It seems you try to use two spark contexts (sparkSession and sc) which is why you get an error. – Shaido Jul 30 '18 at 08:57
  • 1
    Indeed @Shaido the OP is ofc falling in the lazy computation trap but for the time being, the code posted doesn't cause the error given. – eliasah Jul 30 '18 at 09:39

1 Answers1

0

In Spark 2.0 you need to refer the spark context through spark session. You can create empty dataframe as below. It worked for me.

sparkSession.createDataFrame(sparkSession.sparkContext.emptyRDD[Row], baseSchema)

Hope it helps you.

Amar
  • 218
  • 1
  • 4