2

I'm run LBFGS on spark, with 5 features and 10w records, and found treeAggregate this:

enter image description here

We can see the treeAggregte is time-consuming

I have 100 cores, and every job 'treeAggregate at LBFGS.scala:218' has 1w+ tasks

Community
  • 1
  • 1
Dylan Wang
  • 111
  • 8
  • i have ran 'coalesce' on my trainData, reduce partitions equal to cores, and got a big improvement. But why? – Dylan Wang Sep 18 '16 at 06:56

0 Answers0