CPU Consumption of apache spark process

Question

I have a system with 6 physical cores and each core has 8 hardware threads resulting in 48 virtual cores. Following are the setting in configuration files.

spark-env.sh

export SPARK_WORKER_CORES=1

spark-defaults.conf

spark.driver.cores 1

spark.executor.cores 1

spark.cores.max 1

So it means it should only use 1 virtual core but if we see the output from the TOP command, some time, it has very huge spikes e.g the CPU consumption is above 4000 e.g.

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.064t  37312 S  4728  6.4   7:11.30 java
....
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.065t  37312 S  1502  6.5   8:22.75 java
...
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.065t  37312 S  4035  6.6   9:51.64 java
...
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.080t  37312 S  3445  8.1  15:06.26 java
...
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 22581 sbaig     20   0  0.278t 0.082t  37312 S  4178  8.2  17:37.59 java
...

It means, instead of using 1 virtual core, spark is using all available cores in the system so my question is why it is behaving like this? why it is not using only 1 core during execution of job which we set in SPARK_WORKER_CORES property.

I am using spark 1.6.1 with standalone mode.

Any help will be highly appreciated. Thanks Shuja

Did you find a solution for the same, i have exactly the same issues. — Adiant, Jul 04 '18 at 17:24

score -1 · Answer 1 · answered Sep 15 '16 at 10:42

As the per the information you provided, it looks like you are setting the information in spark-defaults.conf file only.

In order to apply this configuration in your spark application, you have to configure these three properties in SparkConf object of code while creating the spark context as shown below.

var conf = new SparkConf()
conf.set("spark.driver.cores","1")
conf.set("spark.executor.cores","1")
conf.set("spark.cores.max","1")

Or if you are submitting the application using the spark-submit CLI then you can use the --driver-cores, --executor-cores and --conf spark.cores.max=1 options while running application.

if you are mentioning in spark-defaults.conf then there is no need to repeat these properties anywhere else. The setting you describing here, is used when you want to override the default properties. Hope I made it clear to you. — David, Sep 22 '16 at 13:23

CPU Consumption of apache spark process

1 Answers1