2

I want to log all the error messages in the driver machine. How to do this efficiently.

vindev
  • 2,240
  • 2
  • 13
  • 20

3 Answers3

0

Basically you need a Custom Logger to write all your application specific logs to separate file instead of writing in YARN logs.

https://mapr.com/blog/how-log-apache-spark/

The above URL says clearly , how you can implement Custom logger mechanism in Apache Spark.

Shankar
  • 8,529
  • 26
  • 90
  • 159
  • Note that a worker node - where your attempted to log the message - can be physically a different machine from where you submitted the program and where the Driver program is running. The issue mentioned here is how to collect messages to the driver, so you have them consolidated in one location. – YoYo Apr 22 '17 at 22:32
0

You can put a try/catch block around your code and print the stack-trace.

 try {

    // your code

  } catch {
    case e: Exception => {
      e.printStackTrace()
    }
Sanchit Grover
  • 998
  • 1
  • 6
  • 9
  • Thanks @Snchit Grover, but this will print error message in the worker node. I need to collect these errors and throw it to driver machine. – vindev Apr 10 '17 at 09:10
  • It would catch exception from your executors and print it on your driver node. Can you try this out? – Sanchit Grover Apr 10 '17 at 09:32
  • Exactly this will print error message on worker node, but I don't what to check each worker node. I want to print all the error message on driver node. – vindev Apr 10 '17 at 09:36
  • I am sorry for the confusion. It would collect exception from the executors and print on the driver. Please try it once. – Sanchit Grover Apr 10 '17 at 09:38
  • Hi I tried to look for exception in master log file but couldn't find one. I'm submitting job using this command `spark-submit --driver-java-options "-Dlog4j.configuration=file:/path/log4j_RequestLogDriver.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/path/log4j_RequestLogWorker.properties" --master spark://IP:7077 --deploy-mode client --class ClassName sample.jar` and then searched for logs in the file path specified in log4j_RequestLogDriver.properties file. – vindev Apr 10 '17 at 12:18
0

Depending on the amount of messages to be collected, you could consider doing this using an Accumulator. The built-in ListAccumulator seems to be a prime candidate.

YoYo
  • 9,157
  • 8
  • 57
  • 74