How can I combine multiple RDD[(String, Double, Double)] into one RDD?

Question

for(i <- 1 to 20) {
    val tmp = rdd.sample(true, 1)
    val rdd2 = resampledData(tmp)
    }

I would like to combine every rdd2 in the loop in single rdd. The type of rdd2 is RDD[(String, Double, Double)] and how can I initialize the new vriable if required?

Possible duplicate of [how to merge two RDD to one RDD](https://stackoverflow.com/questions/41120341/how-to-merge-two-rdd-to-one-rdd) — alejandrogiron, Apr 23 '18 at 00:42
I think it is not duplicate as actually I don't have two RDDs already initialized. I have only one RDD not initialized and I would make recursive merge on this single RDD. — Khaled Omar, Apr 23 '18 at 01:18
val combined = (1 to 20).map(x => resampledData(rdd.sample(true, 1))).reduce(_ union _) — Khaled Omar, Apr 23 '18 at 01:38

score 0 · Answer 1 · answered Apr 23 '18 at 02:31

0

val combined = (1 to 20).map(x => resampledData(rdd.sample(true, 1))).reduce(_ union _)

answered Apr 23 '18 at 02:31

Khaled Omar

27
2

How can I combine multiple RDD[(String, Double, Double)] into one RDD?

1 Answers1