We have a Spark Streaming job, inside DStream foreachRDD method, i am creating a SQLContext, the reason that i am creating SQLContext inside foreachRDD method instead of outside is, when i enable check-pointing it says SQLContext is not serializable.
So i created inside foreachcRDD method, then its working fine.
But i want to close this SQLContext properly at the end of foreachRDD method. How can i do that?
Sample code:
dStream.foreachRDD(rdd =>
{
val sparkContext = rdd.sparkContext
val sqlContext = new HiveContext(sc)
//convert rdd to DF using sqlContext and process it
......
......
......
......
//finally close the sqlContext
sqlContext = null
}
})