Apache Spark Dataframe Groupby agg() for multiple columns

Question

I have a DataFrame with 3 columns i.e. Id, First Name, Last Name

I want to apply GroupBy on the basis of Id and want to collect First Name, Last Name column as list.

Example :- I have a DF like this

+---+-------+--------+
|id |fName  |lName   |
+---+-------+--------+
|1  |Akash  |Sethi   |
|2  |Kunal  |Kapoor  |
|3  |Rishabh|Verma   |
|2  |Sonu   |Mehrotra|
+---+-------+--------+

and I want my output like this

+---+-------+--------+--------------------+
|id |fname           |lName               |
+---+-------+--------+--------------------+
|1  |[Akash]         |[Sethi]             |
|2  |[Kunal, Sonu]   |[Kapoor, Mehrotra]  |
|3  |[Rishabh]       |[Verma]             |
+---+-------+--------+--------------------+

Thanks in Advance

score 16 · Accepted Answer · answered Mar 17 '17 at 06:34

16

You can aggregate multiple columns like this:

df.groupBy("id").agg(collect_list("fName"), collect_list("lName"))

It will give you the expected result.

answered Mar 17 '17 at 06:34

himanshuIIITian

5,985
6
50
70

1

thanks man, easy and clean, still saving tone of hours doing this manually in python (in my case) – Marian Rick Apr 11 '21 at 16:52
I'm glad that you found this answer helpful!. – himanshuIIITian Apr 12 '21 at 04:58

Apache Spark Dataframe Groupby agg() for multiple columns

1 Answers1

Linked