0

I'm wondering how to go about submitting spark "jobs" to a single application (allowing for sharing of RDD work but also code/execution independence of modules). I've seen this spark-jobserver formally at ooyala, but I noticed it doesn't yet support python. Is this the common route taken for this use case in scala/java? Or am I going down the wrong route here?

I've also seen a less popular pyspark-jobserver and open issues on the main spark-jobserver to address python and R.

In order to have a better understanding of the use case of spark-jobserver, I'm also wondering why this functionality wouldn't be supported directly by spark given their detailed job-scheduling framework.

retrocookie
  • 319
  • 2
  • 10
  • Python is not supported by spark-jobserver yet. – noorul Apr 07 '16 at 16:33
  • I believe I mentioned that in my question! I'm hoping for an answer that at least discusses whether this is the right direction. – retrocookie Apr 07 '16 at 16:51
  • In the comment you mentioned about ooyala spark-jobserver (https://github.com/ooyala/spark-jobserver) which is no longer the maintained version. The maintained version is located at https://github.com/spark-jobserver/spark-jobserver. Also I have not answered your question, I was asserting that python support is not their yet in spark-jobserver. This thread got my attention because it has spark-jobserver tag. It doesn't look like a spark-jobserver question, as you already know python is not supported by it. And your question is specific to why spark is not supporting what you want. – noorul Apr 07 '16 at 16:55
  • I did not link to ooyala, and merely specified it was migrated from there. I also already asserted "but I noticed it doesn't yet support python". The reason I tagged spark-jobserver is because I want to learn more about its general use case, and whether or not it makes sense as a future direction for my desired use case. I had a side question to why it isn't directly supported by Spark because I want to have a better understanding of the spark-job-server's use case. I'll edit to make it more clear. – retrocookie Apr 07 '16 at 17:15

0 Answers0