Spark: Extract Values from Output RDD

Question

I am new in Spark programming. I am trying to extract values from RDD as I got the below output from RDD

(CBI10006,(Some(Himanshu Vasani),None))
(CBI10004,(Some(Sonam Petro),Some(8500)))
(CBI10003,(None,Some(3000)))

And I want to extract above value to below one

(CBI10006,Himanshu Vasani,'')
(CBI10004,Sonam Petro,8500)
(CBI10003,'',3000)

And I have tried FlatMap approch as below

joined.flatMap{case(f1,f2) => (f1,(f2._1,f2._2))} but getting a below error

type mismatch;
 found   : (String, (Option[String], Option[String]))
 required: TraversableOnce[?]
    **joined.flatMap{case(f1,f2) => (f1,(f2._1,f2._2))}**

In your case a map would work, unless you want a RDD[List[String]] as a result — Emiliano Martinez, Jan 04 '22 at 10:59
@DhrumilShah no, I mean that you could just call `values` on `joined` to get the answer you want. — Luis Miguel Mejía Suárez, Jan 05 '22 at 03:15

score 2 · Accepted Answer · answered Jan 04 '22 at 18:37

2

Using map():

val data = Seq(("CBI10006", (Some("Himanshu Vasani"), None)), ("CBI10004", (Some("Sonam Petro"), Some(8500))),
  ("CBI10003", (None, Some(3000))))
    
spark.sparkContext
  .parallelize(data)
  .map { case (x, y) => (x, y._1.getOrElse(""), y._2.getOrElse("")) }
  .foreach(println)

// output: 
// (CBI10006,Himanshu Vasani,)
// (CBI10004,Sonam Petro,8500)
// (CBI10003,,3000)

answered Jan 04 '22 at 18:37

Gabio

9,126
3
12
32

Thanks @Gabib it's working....but just for understanding could you suggest me with flatMap option too? – Dhrumil Shah Jan 05 '22 at 02:00
@DhrumilShah there is no point in using `flatMap` for a one to one mapping. Why do you want to use that? – Luis Miguel Mejía Suárez Jan 05 '22 at 03:16

Spark: Extract Values from Output RDD

1 Answers1