To sort a specific column in a DataFrame in SparkR

Question

In SparkR I have a DataFrame data. It contains time, game and id.

head(data)

then gives ID = 1 4 1 1 215 985 ..., game = 1 5 1 10 and time 2012-2-1, 2013-9-9, ... Now game contains a gametype which is numbers from 1 to 10.

For a given gametype I want to find the minimum time, meaning the first time this game has been played. For gametype 1 I do this

data1 <- filter(data, data$game == 1)

This new data contains all data for gametype 1. To find the minimum time I do this

g <- groupBy(data1, game$time)
first(arrange(g, desc(g$time)))

but this can't run in sparkR. It says "object of type S4 is not subsettable".

Game 1 has been played 2012-01-02, 2013-05-04, 2011-01-04,... I would like to find the minimum-time.

Why do you group by time in the first place? Could you provide example input and expected output? — zero323, Sep 15 '15 at 08:32
I have give an example. I groupBy time since I want to use agg-function to it. — Ole Petersen, Sep 15 '15 at 08:58

score 1 · Answer 1 · answered Sep 15 '15 at 09:52

1

By typing

arrange(game, game$time)

I get all of the time sorted. By taking first function I get the first entry. If I want the last entry I simply type this

first(arrange(game, desc(game$time)))

answered Sep 15 '15 at 09:52

Ole Petersen

670
9
21

score 1 · Accepted Answer · answered Sep 15 '15 at 15:48

1

If all you want is a minimum time sorting a whole data set doesn't make sense. You can simply use min:

agg(df, min(df$time))

or for each type of game:

groupBy(df, df$game) %>% agg(min(df$time))

answered Sep 15 '15 at 15:48

zero323

322,348
103
959
935

score 0 · Answer 3 · answered Dec 16 '15 at 14:06

Just to clarify because this is something I keep running into: the error you were getting is probably because you also imported dplyr into your environment. If you would have used SparkR::first(SparkR::arrange(g, SparkR::desc(g$time))) things would probably have been fine (although obviously the query could've been more efficient).

To sort a specific column in a DataFrame in SparkR

3 Answers3