0

I've set up a spark job-server (see https://github.com/spark-jobserver/spark-jobserver/tree/jobserver-0.6.2-spark-1.6.1) in standalone mode.

I've set up some jobs using Scala, every job use the same shared context, but I don't understand how to persist my job's (or context's) logs.

Currently I'm using:

val logger = LoggerFactory.getLogger(getClass)

without any success. I haven't found any docs or examples, is it even possible?

Thank you

Marco Fedele
  • 2,090
  • 2
  • 25
  • 45

1 Answers1

0

I found this helpful in trying to solve the same problem:

https://mapr.com/blog/how-log-apache-spark/

You have to adjust the spark log4j configuration.

Justin Standard
  • 21,347
  • 22
  • 80
  • 89