2

We have 3 compute nodes with 8 cores each and are assigned 30 GB of RAM in our cluster and we are executing performance tests in order to get the optimal performance.

The optimal performance was achieved by considering the following parameters.

master yarn --num-executors 5 --executor-cores 4 --executor-memory 23g

The concern here is how is it working fine with 23*5=115g of memory when we have only 30*3=90g available for our cluster. We have tried by considering the executor memory from 16g to 25g but getting the optimal at 23g only. Is there something missing from our end, would like to understand this concept.

Ram Ghadiyaram
  • 28,239
  • 13
  • 95
  • 121
Sumit Khurana
  • 159
  • 1
  • 10
  • 1
    When you look at the spark UI, what are the sizes of your workers? It could be that one or two workers have 23g and the others have less. – Graham S Sep 21 '16 at 13:04
  • Can we get a snapshot of the YARN Resourcemanager UI for your appication running? – Rakesh Rakshit Sep 22 '16 at 06:01
  • @GrahamS if i understood it clearly, is it like the executor memory may differ per executor from what is specified from our end in the spark submit command. In our case we have explicitly specified it as 23 GB per executor and have mentioned 5 executors. I am afraid, i am not sure on how we can check it using the UI, i was not able to see any option for that. – Sumit Khurana Oct 10 '16 at 08:50
  • @RakeshRakshit Please find below the stats from the Resource Manager :Containers Running:4 Memory Used:77.50 GB Memory Total:90 GB Memory Reserved:25.50 GB vcores Used:13 VCoreS Total:24 VCores Reserved:4 Active Node:3 – Sumit Khurana Oct 10 '16 at 08:50
  • @SumitKhurana Yes, that's what I'm thinking, since you are asking for more resources than you have. [Spark UI](http://spark.apache.org/docs/latest/monitoring.html#web-interfaces) – Graham S Oct 11 '16 at 13:31
  • @GrahamS So is there no way that the spark submit command will itself throw an exception in case the resources being asked for are greater than the available resources? – Sumit Khurana Oct 12 '16 at 08:57

0 Answers0