10

I am learning Spark on AWS EMR. In the process I am trying to understand the difference between number of executors(--num-executors) and executor cores (--executor-cores). Can any one please tell me here?

Also when I am trying to submit the following job, I am getting error:

spark-submit --deploy-mode cluster --master yarn --num-executors 1 --executor-cores 5   --executor-memory 1g -–conf spark.yarn.submit.waitAppCompletion=false wordcount.py s3://test/spark-example/input/input.txt s3://test/spark-example/output21

Error: Unrecognized option: -–conf
marios
  • 8,874
  • 3
  • 38
  • 62
AIR
  • 817
  • 12
  • 24

1 Answers1

15

Number of executors is the number of distinct yarn containers (think processes/JVMs) that will execute your application.

Number of executor-cores is the number of threads you get inside each executor (container).

So the parallelism (number of concurrent threads/tasks running) of your spark application is #executors X #executor-cores. If you have 10 executors and 5 executor-cores you will have (hopefully) 50 tasks running at the same time.

marios
  • 8,874
  • 3
  • 38
  • 62
  • what's the difference between executor-cores and spark.executor.cores? – cozyss Dec 12 '17 at 19:41
  • 1
    @rileyss they are synonyms. The one is used in the configuration settings whereas the other was used when adding the parameter as a command line argument. I just used one of the two on the example here, but there was no particular reason why I choose one over the other. – marios Dec 12 '17 at 20:35