How can I get a progressbar on spark-sql? spark-shell get a nice progress bar like this: [Stage7:===========> (14174 + 5) / 62500]
This progressbar tells what is the total number of executors allocated, how many are running and how many completed. while the length of the progress bar indicates the % completion of the job.
Absent this we have to rely only on the Resource Manager UI and keep refreshing the same.
