0

Spark breaks and executes the code into stages.I also know that it is a progress bar which give information about staging part in Spark. What information do we get from the numbers in the following line in console when the job is running on cluster.

[Stage 133:(4000 + 4) ====> (1 + 1)/13000]

zero323
  • 322,348
  • 103
  • 959
  • 935
Prashant_M
  • 2,868
  • 1
  • 31
  • 24

1 Answers1

2

Progress bar indicates [Stage 133:(4000 + 4) ====> (1 + 1)/13000]

(numCompletedTasks + numActiveTasks) / totalNumOfTasksInThisStage]

  • Number of Completed Tasks = 1

  • Number of Active Tasks = 1

  • Total number of tasks in this stages = 13000

Please refer here for the detail

koiralo
  • 22,594
  • 6
  • 51
  • 72