Is there a way to take the first 1000 rows of a Spark Dataframe?

Question

I am using the randomSplitfunction to get a small amount of a dataframe to use in dev purposes and I end up just taking the first df that is returned by this function.

val df_subset = data.randomSplit(Array(0.00000001, 0.01), seed = 12345)(0)

If I use df.take(1000) then I end up with an array of rows- not a dataframe, so that won't work for me.

Is there a better, simpler way to take say the first 1000 rows of the df and store it as another df?

score 209 · Accepted Answer · edited Nov 19 '20 at 09:51

209

The method you are looking for is .limit.

Returns a new Dataset by taking the first n rows. The difference between this function and head is that head returns an array while limit returns a new Dataset.

Example usage:

df.limit(1000)

edited Nov 19 '20 at 09:51

Pau Coma Ramirez

4,261
1
20
19

answered Dec 10 '15 at 16:12

Markon

4,480
1
27
39

4

Just for the record, I used .limit and it returned a Dataset instead of DataFrame (which was not a problem) – hipoglucido Jul 17 '17 at 11:55
2

Thanks for the update. It seems that Spark 2.X has changed quite a bit :) – Markon Jul 17 '17 at 12:45

Is there a way to take the first 1000 rows of a Spark Dataframe?

1 Answers1

Linked