1

I am trying to broadcast a variable within a for loop in spark. During this process, spark is throwing task not serialize-able error. If the same variable is broadcasted out of the for loop, there is no error. Below is code snippet that throws error. Any help is appreciated.

var Final = computedRDD.filter(x => x.Id == uniqueKey(0))

for (partId <- uniqueKey) {
      val FinalBroadcast = sc.broadcast(Final.collect)
      val computeNew = computedRDD.filter(x => x.partId == partId).repartition(executors).mapPartitions(performFinalPass(FinalBroadcast))
      computeNew.collect.forall(x => Final.add(x))
}
Anan
  • 11
  • 1
  • 2

0 Answers0