I have been working on an issue to fix the spark that by whatever reason if one of the worker pods got restarted, the application started malfunctioning. Please check the log from the spark master pod
Initially, the master pod registers all worker pods and executers (each worker pod has its own executor). Then for any reason, if one of the pod got restated it deletes the application id of all executor then all worker pod stops working because of losing their application id.
I wanted to know what's happening in spark internal and if there is any configuration to fix this
