1

I have a spark streaming job with batch Interval of 10 minutes and slides/window of 1 hour, I have activated the dynamic allocation with Spark.

But executors get removed after 1 hours and not after each 10 min of Batch Interval, So I have to wait 1 hour to see executors removed (in case of no data to process).

Is there a way to tell spark to revise excecutors removal after each batch interval instead of Window interval ?

Thanks.

Amar AttilaZz
  • 242
  • 3
  • 5
  • 13

1 Answers1

3

Please check executor idle timeout property settings in job level or default config (spark-defaults.conf)

  • spark.dynamicAllocation.executorIdleTimeout
  • spark.dynamicAllocation.cachedExecutorIdleTimeout

Refer this link for more details.

Ravikumar
  • 1,121
  • 1
  • 12
  • 23