How to transform a item in RDD(Spark) to many items in RDD?

Question

I get a RDD with 'Scan' action from 'HBase'. Each item in this RDD is like: x1, y1, y2, y3... So the items in this RDD are like(each line is a row result of scan action):

x1, y1, y2, y3
x2, y1, y4, y8, y9
x3, y5
......
xn, y1, y6, y100

I want transform this RDD to another RDD like:

x1, y1
x1, y2
x1, y3
x2, y1
x2, y4
...
xn, y1
xn, y6
xn, y100

How can I do this transform?

I assume the 2nd item is a typo...and should be x1 instead of x2 — Justin Pihony, May 20 '16 at 02:39

score 2 · Answer 1 · answered May 20 '16 at 02:09

2

"flatMap that sh*t":

rdd.flatMap(x => {
  val key = x.head
  x.tail.map(y=>(key,y))
})

answered May 20 '16 at 02:09

Justin Pihony

66,056
18
147
180

No, it's not simply combine the head with each item in the tail. If you observe the last few examples, the 2nd item is computed by `f(xn)`. – David S. May 20 '16 at 02:23
@davidshen84 What makes you think that? – Justin Pihony May 20 '16 at 02:38
Sorry, the 2nd item is a typo. I have edited it.@davidshen84 – York May 20 '16 at 02:50

How to transform a item in RDD(Spark) to many items in RDD?

1 Answers1