1

I have configured Spark jobserver to run on YARN. I am able to send spark jobs to YARN but even after the job finishes it does not quit on YARN

For eg: I tried to make a simple spark context. The context is reflecting in jobserver but YARN is still running the process and is not quieting I have to manually kill the tasks.

Yarn Job

Yarn Job

Spark Context Spark Context

Job server reflects the contexts but as soon as I try to run any task in it Job server give me an error

{
  "status": "ERROR",
  "result": "context test-context2 not found"
}

My Spark UI is also not very helpful

Spark Ui

Ashmeet Singh
  • 344
  • 1
  • 5
  • 17
  • Can you provide more information like, info from spark-jobserver log at the time of execution? – noorul Jan 30 '16 at 06:38

0 Answers0