calling SPARK SQL inside map function

Question

In my code I am have a requirement where I need to call spark sql for each of the rows of a dataset.

Now, spark sql requires SparkSession inside map function, which is not possible to pass as a broadcast Variable.

So, is there anyway to call Spark SQL inside a map function?

I have checked online but I was not able to find any information related to same.

I am using Java as a Programming language for SPARK.

SPARK VERSION : 2.3.0

Thanks in advance.

Can you write your SQL-function that you want to execute? It will be much easier to propose you a solution you need. — ruslangm, Feb 07 '19 at 10:09
@ruslangm I want to fire simple SELECT queries inside map function of SPARK. — A Learner, Feb 21 '19 at 12:08

score -1 · Answer 1 · answered Feb 07 '19 at 10:00

Map applies a function on every item of a Dataset and returns another Dataset. What you need here is to iterate on every item with a org.apache.spark.api.java.function.ForeachFunction. Then you can execute spark sql for each item.

Example:

data.foreach((ForeachFunction<Row>) row -> System.out.println(row));

calling SPARK SQL inside map function

1 Answers1