0

Implementing checkpointing when spark streaming job is diretcly submitted to spark seems straight forward . We are a facing quite some complexities when we need to the same when the streaming job is submitted using Spark Job server..any pointers/reference could be quite helpful. Can anybody help?

Sundaresan Krishnamurthy
  • 1
  • What is the difference in two approaches? – noorul Apr 07 '17 at 08:25
  • @noorul when you submit a spark streaming job using SJS you do not create the StreamingContext in the job implementation. but receive it as a parameter into the overridden runJob method https://github.com/spark-jobserver/spark-jobserver/blob/master/job-server-extras/src/main/scala/spark/jobserver/StreamingTestJob.scala – Sundaresan Krishnamurthy Apr 08 '17 at 11:52
  • If you look at the implementation it is actually returning a StreamingContext https://github.com/spark-jobserver/spark-jobserver/blob/master/job-server-extras/src/main/scala/spark/jobserver/context/StreamingContextFactory.scala#L18 – noorul Apr 08 '17 at 15:33
  • @noorul I am unsure how it is helpful. Can you add more details? – Sundaresan Krishnamurthy Apr 19 '17 at 11:30

0 Answers0