3

I've a job running which shows the Event Timeline as follows, I am trying to guess the gaps between these single lines, they seem to be parallel but not immediately sequencial with other stages...

Any other insight from this, and what is the cluster doing during these gaps?enter image description here

Aakash Basu
  • 1,689
  • 7
  • 28
  • 57

2 Answers2

3

Without any code to look at, a blind guess is that during those gaps the driver is busy doing some work. If you are doing a .collect(), or a broadcast(), or any type of local processing in the driver program, then the executors will sit idle, waiting to have work assigned to them.

Travis Hegner
  • 2,465
  • 1
  • 12
  • 11
0

Note that in a visualization you see tasks from a table below it. If you change a paging size or a sorting of the table, you can see the actual pattern.

mishka
  • 2,027
  • 2
  • 20
  • 30