I have a kubernetes cluster where I try to run a spark example application (spark-pi).
I have setup a service account = spark
Scenario
When I do a spark-submit from the command line like below, I am getting a container error.
bin/spark-submit --master k8s://https://IP:port --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=2 --conf spark.kubernetes.container.image=spark:latest --conf spark.kubernetes.container.image.pullPolicy=Never --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/spark/examples/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar 10
Expectation:
Get the value of pi = 3.14 printed in the logs
Actual result:
The driver pod is not coming up due to an error scenario. when I do a describe on the pod, I see the below:
$ kubectl describe pods spark-pi-driver
"applicationState": {
"state": "FAILING",
"errorMessage": "driver container failed with ExitCode: 1, Reason: Error"
},
Events:none
I don't see any events as well
Is there a better way to troubleshoot this scenario ? Please help.
