2

I am using a fair scheduler along with yarn

spark.scheduler.pool is a local property to set to spark context while executing jobs, for a configured pool.

like..

val sc: SparkContext = ???
sc.setLocalProperty("spark.scheduler.pool", "myPool")

I was just wondering is there any possibility to set this local property from spark-submit command like -Dspark.scheduler.pool=xxx

himanshuIIITian
  • 5,985
  • 6
  • 50
  • 70
user3190018
  • 890
  • 13
  • 26

1 Answers1

0

Yes, spark.scheduler.pool can be set to any value from spark-submit using the following syntax -

spark-submit --master local --conf spark.scheduler.pool=myPool --class classpath jarfile

I hope it helps!

himanshuIIITian
  • 5,985
  • 6
  • 50
  • 70
  • this is not working as expected. pools are configured and I gave the property `--conf` from `spark-submit` not `setLocalProperty` but its taking default pool. i am using client and cluster mode some times not local – user3190018 May 04 '18 at 20:43
  • @user3190018 Can you please share the output of `sc.conf.getAll`? – himanshuIIITian May 05 '18 at 07:49