I have a single node spark cluster (4 cpu cores and 15GB of memory) configured with a single worker. I can access the web UI and see the worker node. However, I am having trouble submitting the jobs using spark-submit. I have couple of questions.
- I have an uber-jar file stored in the cluster. I used the following command to submit a job
spark-submit --class Main --deploy-mode cluster --master spark://cluster:7077 uber-jar.jar. This starts the job but fails immediately with the following log messages.
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/11/13 01:19:47 INFO SecurityManager: Changing view acls to: admin
19/11/13 01:19:47 INFO SecurityManager: Changing modify acls to: admin
19/11/13 01:19:47 INFO SecurityManager: Changing view acls groups to:
19/11/13 01:19:47 INFO SecurityManager: Changing modify acls groups to:
19/11/13 01:19:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(admin); groups with view permissions: Set(); users with modify permissions: Set(admin); groups with modify permissions: Set()
19/11/13 01:19:48 INFO Utils: Successfully started service 'driverClient' on port 46649.
19/11/13 01:19:48 INFO TransportClientFactory: Successfully created connection to cluster/10.10.10.10:7077 after 37 ms (0 ms spent in bootstraps)
19/11/13 01:19:48 INFO ClientEndpoint: Driver successfully submitted as driver-20191113011948-0010
19/11/13 01:19:48 INFO ClientEndpoint: ... waiting before polling master for driver state
19/11/13 01:19:53 INFO ClientEndpoint: ... polling master for driver state
19/11/13 01:19:53 INFO ClientEndpoint: State of driver-20191113011948-0010 is FAILED
19/11/13 01:19:53 INFO ShutdownHookManager: Shutdown hook called
19/11/13 01:19:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-4da02cd2-5cfc-4a2a-ad10-41a594569ea1
what am I doing wrong and how do I correctly submit the job.
- If my uber-jar file is in my local computer, how do I correctly use spark-submit to submit a spark job using the uber-jar file to the cluster from my local computer. I've experimented running spark-shell in my local computer by pointing to the standalone cluster using
spark-shell --master spark:\\cluster:7077. This starts a spark shell in my local computer and I can see (in the spark web UI) the worker gets memory assigned to it in the cluster. However, if I try to perform a task in the shell I get the following error message.WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
