Spark convert single column into array

Question

How can I convert a single column in spark 2.0.1 into an array?

+---+-----+
| id| dist| 
+---+-----+
|1.0|2.0|
|2.0|4.0|
|3.0|6.0|
|4.0|8.0|
+---+-----+

should return Array(1.0, 2.0, 3.0, 4.0)

A

import scala.collection.JavaConverters._ 
df.select("id").collectAsList.asScala.toArray

fails with

java.lang.RuntimeException: Unsupported array type: [Lorg.apache.spark.sql.Row;
java.lang.RuntimeException: Unsupported array type: [Lorg.apache.spark.sql.Row;

Possible duplicate of [Spark Dataframe groupby with agg performing list appending](http://stackoverflow.com/questions/34202997/spark-dataframe-groupby-with-agg-performing-list-appending) — mrsrinivas, Nov 12 '16 at 04:46

cheseaux · Accepted Answer · 2016-11-21T20:48:35.463

8

Why do you use JavaConverters if you then re-transform the Java List to a Scala List ? You just need to collect the dataset and then map this array of Rows to an array of doubles, like this :

df.select("id").collect.map(_.getDouble(0))

edited Nov 21 '16 at 20:48

answered Nov 10 '16 at 14:52

cheseaux

5,187
31
50

4

`collect()` on dataframe is not a scalable way. – anji_rajesh Nov 21 '16 at 13:12
4

Who talked about scalability here ? – cheseaux Aug 17 '20 at 07:10

mrsrinivas · Answer 2 · 2016-11-14T17:54:29.033

6

I'd try something like this with dataframe aggregate function - collect_list() to avoid memory overhead on the driver JVM. With this approach only selected column values will be copied to driver JVM.

df.select(collect_list("id")).first().getList[Double](0)

This returns java.util.List[Double].

edited Nov 14 '16 at 17:54

answered Nov 11 '16 at 04:59

mrsrinivas

34,112
13
125
125

Spark convert single column into array

2 Answers2