Execute multiple Spark jobs

Question

I am running a Spark job with following cluster and application configuration:

Total Node: 3 Master Node Memory 7.5GB, 2 Cores Worker Node1, Memory 15GB, 4 Cores Worker Node2, Memory 15GB, 4 Cores

Application Configuration:

--master yarn --num-executors 2 --executor-cores 2 --executor-memory 2G

I am trying to submit multiple jobs in same time with same user, however I see only first two submitted jobs are executing and third has to wait with following warring.

19/11/19 08:30:49 WARN org.apache.spark.util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
19/11/19 08:30:49 WARN org.apache.spark.util.Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.

I found that SparkUI is being created for every submitted job, and my cluster is accepting only two job a time. Further I observed that it picked up the third job on port 4042 once first submitted job finished the execution. What could be wrong with my cluster that it is accepting only two job at a time?

Here is my Spark Session Code:

val spark: SparkSession = {
  val sparkSession = SparkSession
    .builder()
    .appName("Data Analytics")
    .config("spark.scheduler.mode", "FAIR")
    //.master("local[*]")
    .getOrCreate()
  sparkSession
}

Further my questions are: Why SparkSession is creating SparkUI for each job and how we can solve this problem. Is there any way to use same Session for multiple jobs.

It could be that the resources are not enough for 3 jobs. Try reducing `executor-cores` to 1 and see if it works. — Shaido, Nov 19 '19 at 09:53

Emiliano Martinez · Answer 1 · 2019-11-19T10:50:10.913

There are several things that you have to take into account: Once you execute spark-submit a Spark application is created(client-mode) and a new driver is created and a new port is used for the driver console using the port 4040. That´s the reason of the warning, because you are trying to create another application and another driver, but the port 4040 is already used so it tries to use the 4041. A Spark job is not a Spark Application, is an execution that corresponds to a Spark action, so depending of the number of actions that your program executes the number of jobs that will be spawn.

In your case you are trying to create two executors with two cores, in other words, you are trying to create two JVMs with two cores each, apart of the driver. Because you are using Yarn it will try to provide the 4 cores for each of your applications and one for each driver.

For more info check this link: https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-scheduler-ActiveJob.html

If the deploy-mode is client then your explanation somewhat fits, if its cluster mode then its something else. — sumitya, Nov 19 '19 at 10:52
Seems partially correct. I increased 6 core for master node with following configuration. `--master yarn --num-executors 1 --executor-cores 1 --executor-memory 2G` now it is accepting 3 jobs. — Sandeep Singh, Nov 20 '19 at 11:31

Execute multiple Spark jobs

1 Answers1