Application job submission with out duplication

Question

We are using DataStax Spark 6.0. We are submitting jobs using crontab to run every 5 mins. We wrote script to find if it is running to avoid duplicate submission of same application. Is there a way to stop job submission or keep job in Queue in Spark level, to avoid duplicate jobs with same application.

Thanks Rakesh

I tried using Crontab only

score 0 · Answer 1 · answered Jun 27 '19 at 10:01

0

You can use oozie to shedule your spark job .

answered Jun 27 '19 at 10:01

Firas Sghari

56
3

Hi We are using DataStax. We cant use oozie. My question is can we use something with in Spark? – Hyder Tom Jun 27 '19 at 15:34

Application job submission with out duplication

1 Answers1