Create Spark RDD or dataframe from Strings coming from InputStream in java

Question

I have a stream of Strings in java. That is coming from a csv file on some other machine. I am creating an InputStream and reading csv file line by line from BufferedReader in java as follows.

        //call a method that returns inputStream 


        InputStream stream = getInputStreamOfFile();

        BufferedReader lineStream = new BufferedReader(new InputStreamReader(stream));

        while ((inputLine = lineStream.readLine()) != null) {
            System.out.println("******************new Line***********");
            System.out.println(inputLine);
        }
        lineStream.close();
        stream.close();

Now, I want to create a spark RDD or DataFrame from this.

one solution is, I keep creating new RDD at each line and maintain globle RDD and continue doing union of RDDs. Is there any other solution ?

Note : this file is not on the same machine. It is coming from some remote storage. I do have the HTTP URL of the file.

Keep appending to a buffer and then create the RDD once the csv has finished streaming? Or stream the csv to a file system you can access and then create a dataframe there from that file? — philantrovert, Oct 04 '17 at 12:57
Why can't you read the file itself with sparkContext.textFile("filepath") — Avishek Bhattacharya, Oct 04 '17 at 15:49
@AvishekBhattacharya I don't have the filesystem url. Like we have in amazon s3 as s3n:// . I have http url of the file. Its in minio.io bucket. — KrunalParmar, Oct 04 '17 at 17:21
If you can download here is the solution given https://stackoverflow.com/questions/29741082/how-to-access-a-web-url-using-a-spark-context — Avishek Bhattacharya, Oct 04 '17 at 17:30
@AvishekBhattacharya Yes. This is what I want. But I can not find java version for the same. — KrunalParmar, Oct 09 '17 at 05:40

score 1 · Answer 1 · answered Nov 12 '19 at 19:14

If the contents of the inputStream fits in memory, we can use the following:

private static List<String> displayTextInputStream(InputStream input) throws IOException {
    // Read the text input stream one line at a time and display each line.
    BufferedReader reader = new BufferedReader(new InputStreamReader(input));
    String line = null;
    List<String> result = new ArrayList<String>();
    while ((line = reader.readLine()) != null) {
        result.add(line);
    }
    return result;
}

Now we can convert the List<String> to corresponding RDD.

S3Object fullObject = s3Client.getObject(new GetObjectRequest("bigdataanalytics", each.getKey()));
                            List<String> listVals = displayTextInputStream(fullObject.getObjectContent());
                            JavaRDD<String> s3Rdd = sc.parallelize(listVals);

Create Spark RDD or dataframe from Strings coming from InputStream in java

1 Answers1