2

I'm using Spark streaming 1.6 which uses kafka as a source

My input arguments are as follows:

num-executors    5
num-cores        4
batch Interval  10 sec
maxRate         600
blockInterval   350 ms

Why does some of my executors start later than another ??

enter image description here

Vadym B.
  • 681
  • 7
  • 21

1 Answers1

2

That's not executors' start time, but tasks' start time.

This is most likely due to locality scheduling. Spark delayed the start of a task to find the best executor to run that task on. Check the configuration "spark.locality.wait" in Spark's documentation for further details.

Kien Truong
  • 11,179
  • 2
  • 30
  • 36