How can I union all the DataFrame in RDD[DataFrame] to a DataFrame without for loop using scala in spark?

Question

val result is a spark DataFram and its column is [uid: Int, vector: Vector]. But the type of recomRes is RDD[DataFrame], how can I map union all the result in recomRes to a DataFrame?

val recomRes = result.rdd.map(row => {
    val uid = row.apply(0)
    val vec = row.getAs[Vector](1)
    brp
       .approxNearestNeighbors(vectors, vec, 5)
       .withColumn("uid", lit(uid))
       .select("uid", "aid", "distCol")
}

I have tried for loop to deal with, but very very slow.

score 0 · Accepted Answer · answered Jun 22 '18 at 08:06

0

Use toDF() method after map.

You need to import sqlContext.implicits._

answered Jun 22 '18 at 08:06

Chandan Ray

2,031
1
10
15

I have fixed it by recomRes.reduce((x,y) => x.union(y)) – 许传华 Jun 22 '18 at 09:58
But I tested the program which threw NullPointerException, and I don't know why – 许传华 Jun 22 '18 at 09:59

How can I union all the DataFrame in RDD[DataFrame] to a DataFrame without for loop using scala in spark?

1 Answers1