1

quick question on calling an object and running it from main just need the last command val op = df.coalesce(1).write.mode("overwrite").format("csv").save("report")to be able called from a main class

how do we intilize an instance of object and run to get to write to csv?

  def run(implicit context: Context): Unit = {
      val timer = Timer.start()


      //not working
      val newRep = Report_Adhoc
      val d = newRep.tab.toDF()
      val op = d.coalesce(1).write.mode("overwrite").format("csv").save("report")


println(s"pipeline complete in [${timer.elapsedTime()}]")

}

Main class is this ^ but this throws null point exception

object Report_Adhoc extends App with TransientLogger{

    // code not including too verbose

val tab = 
  counts
   .filter(c => c._1.id.nonEmpty && c._2.id.nonEmpty)
  .map(c => (c._1, c._2, c._3, c._3.values.sum))
  .sort($"_4".desc)
  .map(count =>
    row(
      count._1.id, count._1.label,
      count._2.id, count._2.label,
      count._3(CITE), count._3(CROSS), count._3(MANUAL),
      count._3(RECIPROCAL), count._3(TRANSITIVE), count._3(FAMILY),
      count._4
    )

  )

  val df = tab.toDF()

  val op = df.coalesce(1).write.mode("overwrite").format("csv").save("report")

}

dedpo
  • 482
  • 11
  • 30
  • Possible duplicate of [In Scala; should I use the App trait?](https://stackoverflow.com/questions/24437423/in-scala-should-i-use-the-app-trait) (TL;DR: do not use App trait for running Spark, declare the main method in the old-fashioned way instead) – Alex Savitsky Feb 21 '18 at 18:24

1 Answers1

-1

take a look at this:

https://www.scala-lang.org/api/2.12.3/scala/DelayedInit.html

https://www.scala-lang.org/api/2.12.3/scala/App.html

Also, don't put underscore in names. don't reuse main class/object. Last but not least, make a minimum reproduction instead of pasting big block of code like this.

Jason Hu
  • 6,239
  • 1
  • 20
  • 41
  • i cut down what i thought i could just now. The links don't help :/ – dedpo Feb 21 '18 at 17:57
  • @dedpo it does if you read carefully `DelayedInit`. you are reusing an `App`, which is a subtrait of `DelayedInit`, which means your fields are not initialized without explicitly call `delayedInit`. as I said, you should not write code like such to begin with. – Jason Hu Feb 21 '18 at 17:58