1

I have a spark application written in Java which runs everyday. I have a requirement to exit the program(stop the spark application) based on a condition. I am currently using System.exit(0) for shutting down the application. But i see the application status shows as FAILED in Yarn UI.

FinalStatus Reported by AM: FAILED

Inside Code:

if(<some condition>){
    System.exit(0);
}

Please suggest how to completely stop/shutdown the spark application internally from the code, where the FinalStatus shows "SUCCESS".

mthmulders
  • 9,483
  • 4
  • 37
  • 54
user1326784
  • 627
  • 3
  • 11
  • 31
  • Are you looking for `spark.stop()`? – Som Aug 21 '20 at 03:56
  • you can try spark.stop to Stop the underlying SparkContext .... but if spark starts executing job then we can't update FinalStatus. FinalStatus always depends on your submitted tasks behavior... i'm assuming is lookup of some particular value if that is the case then keep that value in some distributed database and access it in all spark trasformations and explicitly check that value . – kavetiraviteja Aug 24 '20 at 13:05

0 Answers0