0

I have a spark thrift server,and users can submit spark sql to this server.

I would like I can control the number of executors that will be used to run user's sql.That is, I don't want executors will be given as many as possible(later submitted sql may have to wait)

I would ask whether this could be possible.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Tom
  • 5,848
  • 12
  • 44
  • 104

1 Answers1

0

This controls max and min number of executors, but statically:

spark.dynamicAllocation.maxExecutors
spark.dynamicAllocation.minExecutors

Dynamically, afaik, it is not possible.

Pavel
  • 424
  • 3
  • 12