-1

I want to kill the spark job programatically. Following is the scenario:

When I kill the spark job using yarn application -kill <app_id> it is getting killed, but if I do the ps -ef | grep <app_name> then it is that spark job entry. How do I make sure it is getting killed from ps -ef as well?

I want to do this programatically since I am doing yarn application -kill through code.

Any help regarding this is appreciable.

Thanks in advance.

Saurabh Deshpande
  • 1,153
  • 4
  • 18
  • 35
  • 2
    Do you really think it work ? Spark job run in cluster mode normally and run over multi nodes then how your ps command can check job running on different node or not. You can only check spark job if it running on your system. Just go with Yarn command which will work correctly. Since all job spark jobs are own by Yarn if Yarn is RM. – Nikhil Suthar Jun 20 '19 at 14:03

1 Answers1

0

You are going to want to use 'ps -ef | grep SparkSubmit' to find any abberant Spark jobs. Then use kill -9 on their PID ID to kill them. DONT KILL THE MAIN SPARK JOB!

//Find all the java jobs
[stack_overflow@stack_overflow ~]$ ps -ef | grep SparkSubmit
stack_overflow  96747  96736 99 11:19 pts/15   00:01:55 /usr/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Dscala.usejavacp=true -Xmx1g -Dderby.system.home=/home/stack_overflow/Spark/ org.apache.spark.deploy.SparkSubmit --conf spark.local.dir=/opt/spark/temp_land/spark-temp --conf spark.driver.extraJavaOptions=-Dderby.system.home=/home/stack_overflow/ --class org.apache.spark.repl.Main --name Spark shell spark-shell
stack_overflow  97410  14952  0 11:20 pts/15   00:00:00 grep --color=auto SparkSubmit
//96747 is the Spark job I forced to become unresponsive
//97410 is the Base Spark Account don't delete
////Run the kill command on the job, only works if you have permissions on that job
[stack_overflow@stack_overflow ~]$ kill -9 96747
//The job is now dead and gone
[stack_overflow@stack_overflow ~]$ ps -ef | grep SparkSubmit
stack_overflow  96190  14952  0 11:17 pts/15   00:00:00 grep --color=auto SparkSubmit
afeldman
  • 492
  • 2
  • 10