I am trying to run the following code to submit a spark application to a kubernetes' cluster:
/opt/spark/bin/spark-submit --master k8s://https://<spark-master-ip>:6443 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=newfrontdocker/spark:v3.0.1-j14 local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar 100
When I try to run this code, the container runs temporarily before I end up getting this particular error message and the container terminates:
container status:
container name: spark-kubernetes-driver
container image: newfrontdocker/spark:v3.0.1-j14
container state: terminated
container started at: 2021-07-17T11:49:46Z
container finished at: 2021-07-17T11:49:48Z
exit code: 101
termination reason: Error
21/07/17 06:49:13 INFO LoggingPodStatusWatcherImpl: Application status for spark-c15c11340f204794b51bf8d79397bf9e (phase: Failed)
21/07/17 06:49:13 INFO LoggingPodStatusWatcherImpl: Container final statuses:
container name: spark-kubernetes-driver
container image: newfrontdocker/spark:v3.0.1-j14
container state: terminated
container started at: 2021-07-17T11:49:46Z
container finished at: 2021-07-17T11:49:48Z
exit code: 101
termination reason: Error
21/07/17 06:49:13 INFO LoggingPodStatusWatcherImpl: Application spark-pi with submission ID default:spark-pi-4ed9627ab44c778d-driver finished
21/07/17 06:49:14 INFO ShutdownHookManager: Shutdown hook called
21/07/17 06:49:14 INFO ShutdownHookManager: Deleting directory /tmp/spark-3d1a7ff3-1dc9-4db0-acfa-ed52706122b6
What is the exit code 101 and what do I need to do to fix this issue so I run spark apps on the kubernetes' cluster?
