How to read multiple line elements in Spark?

Question

When you read a file in Spark using sc.textfile, it gives you elements, where each element is a separate line. However, I want each element to consist of N number of lines. I can't use delimiters either because there is none in that file. So, how can I make spark give me multiple line elements?

And I'm interested in doing so using the NLineInputFormat class. Is that possible to do so in Spark? I can see examples of that for MapReduce, but I don't have any clue how that would translate to in Spark.

Possible duplicate of [How to process multi line input records in Spark](http://stackoverflow.com/questions/27541637/how-to-process-multi-line-input-records-in-spark) — Mateusz Dymczyk, Mar 17 '16 at 05:46
@MateuszDymczyk I want to do it using the NLineInputFormat class. That is how my question is different. — pythonic, Mar 17 '16 at 06:24

score 2 · Accepted Answer · answered Mar 17 '16 at 07:24

2

Yes, if you are getting the files from hadoop. You should be able to do it like this:

val records = sc.newAPIHadoopRDD(hadoopConf,classOf[NLineInputFormat],classOf[LongWritable],classOf[Text])

Here's the API doc.

answered Mar 17 '16 at 07:24

Mateusz Dymczyk

14,969
10
59
94

Thanks for the answer. But how do you specify the number of lines? – pythonic Mar 17 '16 at 07:36
In the hadoop config, just like if you were using it in MapReduce – Mateusz Dymczyk Mar 17 '16 at 07:37
1

Would something like this be correct? val c = new Configuration(sc.hadoopConfiguration); c.set("lineinputformat.linespermap", 1000); – pythonic Mar 17 '16 at 07:40
Yeah iirc that worked for me. Try it out and comment if something changed and it doesnt work. – Mateusz Dymczyk Mar 17 '16 at 07:45

How to read multiple line elements in Spark?

1 Answers1

Linked