0

I have looked at the answer to

Why is Spark detecting 8 cores, when I only have 4?

And it doesn't seem to explain the following scenario: I am setting the spark.executor.cores at 5. I have spark.dynamicAllocation.enabled set to true. According to the Spark History Server, my 10 node cluster is running 30 executors, indicating that spark is using 3 executors per node. This seems to suggest that 15 cores are available (3 executors x 5 cores) per node. The specs for an m4.xlarge instance are 4 vCPUs with 16 GB of memory. Where are these extra cores coming from?

Note: I am setting spark.executor.memory at 3g and yarn.nodemanager.resource.memory-mb at 12200.

B. Smith
  • 1,063
  • 4
  • 14
  • 23
  • You have 2, not 4... https://aws.amazon.com/ec2/virtualcores/ – OneCricketeer Oct 24 '17 at 21:09
  • @cricket_007 I believe that is innaccurate: https://aws.amazon.com/ec2/instance-types/. Also the lscpu command shows 4 CPUs. – B. Smith Oct 24 '17 at 22:48
  • Cores are not the same as vCPU, the latter being **virtual**, whereas cores are a physical limitation of the server, from which EMR scales up and down, so there's no ideal way to know what the actual server resources are acquired – OneCricketeer Oct 24 '17 at 23:26

0 Answers0