0

I'm trying to enable Spark history server in single standalone mode on my Mac. I have a spark-master service running and am able to run jobs. I also have a history-server service running on localhost. I'm able to view it in my browser but there are no records of any applications on it. I can only assume I've missed some configuration bur I don't know what. Can anybody please advise how to configure history server to work with local standalone Spark?

Here is my configuraiton:

# spark.master spark://master:7077 
spark.eventLog.enabled true 
# spark.eventLog.dir hdfs://namenode:8021/directory 
# spark.serializer org.apache.spark.serializer.KryoSerializer 
# spark.driver.memory 5g 
# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
pac
  • 491
  • 1
  • 7
  • 30
  • Have you enable event logging of a Spark application using `spark.eventLog.enabled`? Do you use a custom `spark.eventLog.dir`? What's the version of Spark? – Jacek Laskowski Jan 10 '18 at 14:33
  • Yes. That's probably the only thing I've configured in relation to this. – pac Jan 10 '18 at 15:45
  • How did you turn that configuration on? Show the entire command line. Do you have any files in `spark.eventLog.dir`? – Jacek Laskowski Jan 10 '18 at 17:13
  • I haven't it configured. Here's my configuration: # spark.master spark://master:7077 spark.eventLog.enabled true # spark.eventLog.dir hdfs://namenode:8021/directory # spark.serializer org.apache.spark.serializer.KryoSerializer # spark.driver.memory 5g # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three" – pac Jan 12 '18 at 22:00
  • Can you edit your question and add the configuration (it's hardly readable as a comment)? You said "I haven't it configured." -- you **have not**? Do you have `spark.eventLog.enabled` or not? Do you have any files in `spark.eventLog.dir`? – Jacek Laskowski Jan 13 '18 at 11:47
  • Do you have any files in `spark.eventLog.dir` (defaults to `file:///tmp/spark-events`)? – Jacek Laskowski Jan 13 '18 at 15:43
  • Sorted now. Had a typo in spark-events. Once I fixed the directory name I could start the server. – pac Jan 22 '18 at 08:55

1 Answers1

0

Directory spark-events was spelled incorrectly. I had spark-event. Once I renamed it I was able to view the job logs.

pac
  • 491
  • 1
  • 7
  • 30