How to extract a range of rows from sequence file in Spark?

Question

Suppose I have a very large sequence file, but I only want to work with first 1000 rows locally. How can I do that?

Currently my code looks like this

JavaPairRDD<IntWritable,VectorWritable> seqVectors = sc.sequenceFile(inputPath, IntWritable.class, VectorWritable.class);

I have it gives me an error saying it returns an array, not a RDD — user3086871, Jan 19 '17 at 18:39
Yeah, it returns you a list of 1000 elements and you can then work with it? What do you expect it to return you ? — Amit, Jan 19 '17 at 18:42
Please note that "take" is an action and not transformation hence you get the List. Applying a transformation to a RDD yields in another RDD. — Amit, Jan 19 '17 at 19:30

score 1 · Accepted Answer · answered Jan 19 '17 at 20:05

1

what you should do is parallelize of these array:

JavaPairRDD<IntWritable,VectorWritable> RDDwith1000 = sc.parallelize(seqVectors.take(1000));

see simple example here and below:

answered Jan 19 '17 at 20:05

Ronak Patel

1 Answers1