SPARK: replacing foreach with flatmap in spark not working

Question

I have a spark3 application which has JavaPairRDD and called the foreach function to iterate though each JavaPairRDD which is working fine. Problem with foreach is I can't return any value as it's associated with VoidFunction.

I changed foreach to flatmap and it's compiled properly but during runtime flatmap class isn't called and the control breakpoint does't go the class Class_call.

currently I have spark foreach call as:

partitionPairInserts.foreach(new Class_call());

and class defination is :

public class Class_call implements Serializable, VoidFunction<Tuple2<dest, 
Iterable<object>>> {}

  public void call(Tuple2<dest, Iterable<Object>> tuple2) 

{
//control comes here in foreach
}

above implementation works as expected but not returning anything back. so, I changed implementation with flatmap to return some value back as:

     JavaRDD<Row> result = partitionPairInserts.flatMap(new Class_call());

public class Class_call implements FlatMapFunction<Tuple2<dest, Iterable<Object>>,Row>, Serializable {}

   public Iterator<Row> call(Tuple2<dest, Iterable<Object>> tuple2) 

{
//control never reach here.
}

once I change foreach to flatmap in spark3, I am not able to call the Class. How will I able to change the call so that I can return value in java spark3.

SPARK: replacing foreach with flatmap in spark not working

0 Answers0