Spark application is deployed in Standalone cluster mode with supervise enabled.
During high availability testing , when a rack with driver instance is powered off (ungracefully) , spark master didn't know about the killed driver and application and master continuously launch executors for the application for about 15 minutes .
masterlogs (below is logged for 15 minutes)
2018-03-09 18:09:02 INFO org.apache.spark.internal.Logging$class:54 - Launching executor app-20180309175053-0002/5202 on worker worker-20180309171520-10.247.247.191-51426
2018-03-09 18:09:02 INFO org.apache.spark.internal.Logging$class:54 - Removing executor app-20180309175053-0002/5153 because it is EXITED
2018-03-09 18:09:02 INFO org.apache.spark.internal.Logging$class:54 - Launching executor app-20180309175053-0002/5203 on worker worker-20180309171632-10.247.247.156-57784
2018-03-09 18:09:02 INFO org.apache.spark.internal.Logging$class:54 - Removing executor app-20180309175053-0002/5155 because it is EXITED
2018-03-09 18:09:02 INFO org.apache.spark.internal.Logging$class:54 - Launching executor app-20180309175053-0002/5204 on worker worker-20180309123802-10.247.247.121-45652
2018-03-09 18:09:02 INFO org.apache.spark.internal.Logging$class:54 - Removing executor app-20180309175053-0002/5157 because it is EXITED
after 15th minute
2018-03-09 18:09:16 WARN org.apache.spark.internal.Logging$class:66 - Got status update for unknown executor app-20180309175053-0002/5282
2018-03-09 18:09:16 WARN org.apache.spark.internal.Logging$class:66 - Got status update for unknown executor app-20180309175053-0002/5295
2018-03-09 18:09:16 WARN org.apache.spark.internal.Logging$class:66 - Got status update for unknown executor app-20180309175053-0002/5296
2018-03-09 18:09:16 WARN org.apache.spark.internal.Logging$class:66 - Got status update for unknown executor app-20180309175053-0002/5289
2018-03-09 18:09:16 WARN org.apache.spark.internal.Logging$class:66 - Got status update for unknown executor app-20180309175053-0002/5277
executor logs
2018-03-09 18:50:17 INFO org.apache.spark.internal.Logging$class:54 - Asked to kill executor app-20180309180931-0004/50
2018-03-09 18:50:17 INFO org.apache.spark.internal.Logging$class:54 - Runner thread for executor app-20180309180931-0004/50 interrupted
2018-03-09 18:50:17 INFO org.apache.spark.internal.Logging$class:54 - Killing process!
2018-03-09 18:50:17 INFO org.apache.spark.internal.Logging$class:54 - Executor app-20180309180931-0004/50 finished with state KILLED exitStatus 143
I checked in spark master code , could not find much in there .
Any help is appreciated , thanks .
