I am trying to broadcast a variable within a for loop in spark. During this process, spark is throwing task not serialize-able error. If the same variable is broadcasted out of the for loop, there is no error. Below is code snippet that throws error. Any help is appreciated.
var Final = computedRDD.filter(x => x.Id == uniqueKey(0))
for (partId <- uniqueKey) {
val FinalBroadcast = sc.broadcast(Final.collect)
val computeNew = computedRDD.filter(x => x.partId == partId).repartition(executors).mapPartitions(performFinalPass(FinalBroadcast))
computeNew.collect.forall(x => Final.add(x))
}