1

Recently we upgraded EMR release label from emr-5.16.0 to emr-5.20.0, which use Spark 2.4 instead of 2.3.1.

At first, it was terrible. Jobs started to take much more than before. Finally, we set maximumResourcesAllocation to true (maybe it was true by default in emr-5.16) and things start to look better. But some stages are still taking much more than before (while others take less). i.e: Spark 2.4: enter image description here

Spark 2.3.1: enter image description here

Images from application history.

With Spark 2.4, the keyBy operation took more than 10x what it took with Spark 2.3.1.

It is related to the number of partitions. In spark 2.3.1, the number of partitions of the RDD after mapToPair operation was 5580, while in Spark 2.4 was the default parallelism of the cluster (128 because it was running in a 64 cores cluster).

I tried forcing a repartition to 10000 and the keyBy stage ended in only 1.2 minutes. But I think is not a good nor definitive solution.

Is this a known issue? Should I try to set default parallelism to a bigger quantity?

Pedro
  • 21
  • 4
  • Do you know what version of Scala they use? I had some issues with Spark 2.4 and Scala 2.12 but my problems were more "violent" the apps would crash, back to 2.11 it was fine. I am using Java apps, not Scala, – jgp Feb 07 '19 at 22:32
  • Emr 5.20 : 2.11.12; Emr 5.16 : 2.11.8; I use Java apps too. – Pedro Feb 08 '19 at 15:00
  • It seems to be related to the property spark.default.parallelism, which is ... 128 . I'm investigating why with 5.20 that step is using default parallelism instead of the number of partitions of the input RDD . – Pedro Feb 08 '19 at 15:45
  • 2
    I force the parallelism of the keyBy to 10000 and it finished in 1.2 minutes. It's absurd. The thing is I don't want to hardcode paralellism. – Pedro Feb 08 '19 at 16:06
  • Did you adjust the memory the same way? When looking at the screenshots, i see that 2.4 consumes less memory vs. 2.3.1... Do your workers have limited memory? – jgp Feb 08 '19 at 16:24
  • The numbers on the screenshots are the Shuffle read (Total shuffle bytes and records read, which includes data read locally and from remote executors). 5.1 gb in 2.3.1 vs 2.1 in 2.4. I really can't explain the difference... I did not change the memory configuration. It should be more than enough. I've checked gc logs and they looked good. – Pedro Feb 08 '19 at 16:36

0 Answers0