I have to count the repeating values in an array
val arr = Array(1,2,2,3,4,5,5,5)
For example how to count the number of 5s in the array using RDD, Dataframe, Datasets?
Asked
Active
Viewed 182 times
1 Answers
1
If you put your Scala array int into Seq
val arr = Seq(1,2,2,3,4,5,5,5).toDF("num")
val counts = arr.groupBy($"num").agg(count($"num"))
scala> counts.show
+---+----------+
|num|count(num)|
+---+----------+
| 1| 1|
| 3| 1|
| 5| 3|
| 4| 1|
| 2| 2|
+---+----------+
howie
- 2,587
- 3
- 27
- 43
-
ok..ty..for example if I want only one number count like for example 5 then? – Chaitanya Jun 05 '19 at 03:36
-
1arr.filter("num==5").groupBy($"num").agg(count($"num")) – howie Jun 05 '19 at 03:40
-
I wouldn't use groupBy because of the shuffle penalty. I'd use a simple foreachPartition along with Accumulator. – Nir Hedvat Jun 05 '19 at 06:39
-
@NirHedvat, that might depend on how many distinct values are expected. If a really large number, best to keep the results as a DF/RDD? – The Archetypal Paul Jun 05 '19 at 10:47