How to traverse/iterate a Dataset in Spark Java?

Question

I am trying to traverse a Dataset to do some string similarity calculations like Jaro winkler or Cosine Similarity. I convert my Dataset to list of rows and then traverse with for statement which is not efficient spark way to do it. So I am looking forward for a better approach in Spark.

public class sample {

    public static void main(String[] args) {
        JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("Example").setMaster("local[*]"));
        SQLContext sqlContext = new SQLContext(sc);
        SparkSession spark = SparkSession.builder().appName("JavaTokenizerExample").getOrCreate();

        List<Row> data = Arrays.asList(RowFactory.create("Mysore","Mysuru"),
                RowFactory.create("Name","FirstName"));
        StructType schema = new StructType(
                new StructField[] { new StructField("Word1", DataTypes.StringType, true, Metadata.empty()),
                        new StructField("Word2", DataTypes.StringType, true, Metadata.empty()) });

        Dataset<Row> oldDF = spark.createDataFrame(data, schema);
        oldDF.show();
        List<Row> rowslist = oldDF.collectAsList(); 
    }
}

I have found many JavaRDD examples which I am not clear. An Example for Dataset will help me a lot.

When I am doinf oldDF.collectAsList(); for my dataset I am getting illegalArgumentException. Does any one know possible reason — Dhirendra Gautam, Aug 11 '22 at 06:49

score 28 · Accepted Answer · answered Mar 13 '17 at 06:24

28

You can use org.apache.spark.api.java.function.ForeachFunction like below.

oldDF.foreach((ForeachFunction<Row>) row -> System.out.println(row));

answered Mar 13 '17 at 06:24

abaghel

14,783
2
50
66

Thanks for the answer. Is there any way I can access previous element and current element at once? I want to compare a column in prev row and current row and make a decision. Thanks in advance! – Saisumanth Gopisetty Feb 17 '23 at 19:47

score 3 · Answer 2 · answered Jul 30 '19 at 04:16

For old java jdks that don't support lambda expressions, you can use the following after importing:

import org.apache.spark.api.java.function.VoidFunction;

yourDataSet.toJavaRDD().foreach(new VoidFunction<Row>() {
        public void call(Row r) throws Exception {
            System.out.println(r.getAs("your column name here"));
        }
    });

How to traverse/iterate a Dataset in Spark Java?

2 Answers2

Linked