-1

In my code I am have a requirement where I need to call spark sql for each of the rows of a dataset.

Now, spark sql requires SparkSession inside map function, which is not possible to pass as a broadcast Variable.

So, is there anyway to call Spark SQL inside a map function?

I have checked online but I was not able to find any information related to same.

I am using Java as a Programming language for SPARK.

SPARK VERSION : 2.3.0

Thanks in advance.

A Learner
  • 157
  • 1
  • 5
  • 16

1 Answers1

-1

Map applies a function on every item of a Dataset and returns another Dataset. What you need here is to iterate on every item with a org.apache.spark.api.java.function.ForeachFunction. Then you can execute spark sql for each item.

Example:

data.foreach((ForeachFunction<Row>) row -> System.out.println(row));
sgungormus
  • 119
  • 5