0

I need to use yarn application -stop to gracefully stop my Spark application. According to this thread we need to set

yarn.application.admin.client.class.SPARK

first. But (1) how and where to set it?

Also in the thread, some framework implementation needs to be available, and again (2) how and to get it? There's scare information about the guide available.

Would you please give some help?

landau
  • 41
  • 1
  • 6
  • Has anyone have any ideas? – landau Nov 01 '22 at 02:50
  • I don't think above approach is right to stop the Spark Application Gracefully. If you are using Batch job, you can use spark.stop() method. If you want to use Streaming Job, you can use spark.streaming.stopGracefullyOnShutdown parameter or Shutdown Spark Streaming application using marker file – Ranga Reddy Mar 16 '23 at 12:04
  • Hi @RangaReddy, I used Spark Structured Streaming instead of Spark Streaming, so the above config doesn't seem to apply. I guess ensuring the graceful shutdown of the JVM requires YARN to wait for a certain time before removing the container where the JVM is living. That's why I want to handle YARN shutdown. – landau Mar 17 '23 at 08:41
  • Do you have exact steps? If yes please share it here. – Ranga Reddy Mar 20 '23 at 04:42
  • Hi @landau, Could you please share the steps – Ranga Reddy Apr 13 '23 at 13:47

0 Answers0