0

Can I evaluate spark inside scala toolbox? I need to read something with spark within the toolbox, my code looks like this:

object Something extends SparkScript {
  def main(args: Array[String]): Unit = {
    val command =
      s"""
        |import spark.implicits._
        |
        |val ds = spark.read.parquet("$path").as[SomeGeneric]
        |ds.write.csv("$anotherPath")
        |""".stripMargin

    val toolBox = currentMirror.mkToolBox()
    toolBox.eval(toolBox.parse(command))
  }
}

By the way, the SparkScript is a trait that create a SparkSession the toolbox always throw an error that said object read is not a member of spark

Dmytro Mitin
  • 48,194
  • 3
  • 28
  • 66
afghifari
  • 27
  • 6
  • 2
    You need to put everything you want/need to use inside the toolbox, it is like a different scope – Pedro Correia Luís Oct 08 '19 at 16:45
  • @PedroCorreiaLuís so, to make it work, i need to create a sparksession inside the toolbox ?, because i did import the sparksession from `Something` inside the toolbox but it still didn't work – afghifari Oct 08 '19 at 16:50
  • Could you put the trait SparkScript and the imports for reflection, so I could try it out? I never did something like that with Spark @afghifari – Pedro Correia Luís Oct 08 '19 at 17:08
  • @PedroCorreiaLuís i finally figure it out, just like you said, i need to import the sparksession inside the toolbox – afghifari Oct 09 '19 at 04:31
  • @afghifari can you post your final solution, I am looking for something like this, thanks! – sramalingam24 Nov 10 '21 at 00:12

0 Answers0