Here is my issue, now, when I start using the spark shell, it would consume lot of resources and perhaps keep them bounded/held up; there by impacting other parallel running applications.
say for example, i am running some spark-shell commands and accidentally leave the shell open and not close the session, all resources it will keep held up, and all other users wont have anything to work on, unless i close my session
How to fix this issue from yarn perspective.
You may want to set resource pools usage for Yarn in Cloudera. You can allocate some resources to each users. Even if you use all your resources, there will be some available for others users.
If you don't want to split yarn resources between users. You can set SPARK to use dynamic allocation (check spark.dynamicAllocation.enabled property in http://spark.apache.org/docs/latest/configuration.html). So if you leave your spark-shell opened and your job is finished, spark will give back the resource to Yarn. But you can't set the number of executors while using dynamic allocation
Regards, Arnaud
