foreachRDD to pull average number of words & characters for each RDD in J8 Spark Streaming of Twitter API

Question

I'm trying to get the average number of words and characters in each RDD I pull from the Twitter API using spark in Java 8. However, I'm having trouble using streams to achieve this. My code is as follows:

//Create the stream.
JavaReceiverInputDStream<Status> twitterStream = TwitterUtils.createStream(jssc);
//Outputs the text of tweets to a JavaDStream.
JavaDStream<String> statuses = twitterStream.map(Status::getText);
//Get the average number of words & characters in each RDD pulled during streaming.
statuses.foreachRDD(rdd -> {
            long c = rdd.count();
            long wc = rdd.map(s -> s.split(" ").length).reduce(Integer::sum);
            long cc = rdd.map(s -> s.split("").length).reduce(Integer::sum);
            long avgWc = wc / c;
            long avgCc = cc / c;
            System.out.println(wc / c);
            System.out.println(cc / c);
        return avgWc, avgCc;});

The error I'm getting is that the return type expected for foreachRDD is void and my return is a long format.

How can I get around this? Is there another way I need to approach this?

if you want to map each RDD in the stream to something, you should consider `mapPartitions` as operation. — UninformedUser, May 02 '21 at 17:46
Thanks @UninformedUser how would I go about doing such an operation in this context though? — M00N KNIGHT, May 02 '21 at 17:59

score 1 · Accepted Answer · answered May 02 '21 at 20:12

A possible solution would be to use JavaDStream.transform. This functions allows to stay within the SparkStreaming-API:

JavaDStream<String> statuses = ...
JavaDStream<Tuple2<Long, Long>> avgs = statuses.transform(rdd -> {
            long c = rdd.count();
            long wc = rdd.map(s -> s.split(" ").length).reduce(Integer::sum);
            long cc = rdd.map(s -> s.split("").length).reduce(Integer::sum);
            long avgWc = wc / c;
            long avgCc = cc / c;
            //System.out.println(wc / c);
            //System.out.println(cc / c);
            return jssc.sparkContext().parallelize(Collections.singletonList(Tuple2.apply(avgWc, avgCc)));
        }
);
avgs.print();

If it is changed to "new Tuple2.apply..." I believe this code will work yes thank you! — M00N KNIGHT, May 02 '21 at 20:32

Neetesh Kumar Chaurasia · Answer 2 · 2021-05-03T04:36:44.393

There is no possibility to return data if the return type is void. You can create a list outside the "foreachRDD" function and pass the values as shown below:

List<Data> listData=new ArrayList();
statuses.foreachRDD(rdd -> {
            long c = rdd.count();
            long wc = rdd.map(s -> s.split(" ").length).reduce(Integer::sum);
            long cc = rdd.map(s -> s.split("").length).reduce(Integer::sum);
            long avgWc = wc / c;
            long avgCc = cc / c;
            System.out.println(wc / c);
            System.out.println(cc / c);
            Data data=new Data();
            data.setAvgCc(avgCc);
            data.setAvgWc(avgWc);
            listData.add(data);
        });

Data is a class with two variables avgCc and AvgWc as shown below

public class Data {
    long avgWc;
    long avgCc;
    public long getAvgWc() {
        return avgWc;
    }
    public void setAvgWc(long avgWc) {
        this.avgWc = avgWc;
    }
    public long getAvgCc() {
        return avgCc;
    }
    public void setAvgCc(long avgCc) {
        this.avgCc = avgCc;
    }
    public Data(long avgWc, long avgCc) {
        super();
        this.avgWc = avgWc;
        this.avgCc = avgCc;
    }
    public Data() {
        super();
    }
}

Please let me know if this helps. Or you need more clarifications.

Thanks. The code still doesn't seem to work however. What Data library is this you're referencing? — M00N KNIGHT, May 02 '21 at 19:38

foreachRDD to pull average number of words & characters for each RDD in J8 Spark Streaming of Twitter API

2 Answers2