Spark shell error : ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 11

Question

Whenever I start the spark-shell on mapr Sandbox I keep on getting this error

ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 11

Please help me to solve this error.

Answer 1

I faced similar issue and it got resolved after following these steps:

1. Restart you spark-master using this command: maprcli node services -name spark-master -action restart -filter csvc==spark-master

2. Start the worker nodes: /opt/mapr/spark/spark-1.3.1/sbin/start-slaves.sh

3. Now, run the spark-shell: /opt/mapr/spark/spark-1.3.1/bin/spark-shell

Hope this helps !

Answer 2

use hostname command to find host-name and then use command

bin/spark-shell --master spark://your-host-name:7077

Answer 3

I hit this issue when I setup SPARK_JAVA_OPTS for remote debugging, but didn't actually connect my IDE's debugger to the Spark application.

I discovered this when I looked in my Spark UI under workers at the executors stdout/stderr, and noticed the following:

FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)

Answer 4

In favor of googlers who got here cause of the error log: We had the same problem when we tried to start zeppelin outside of the cluster, We discover that the problem was that there were no connection between the workers and the driver process.

Bottom line: Open all tcp ports between workers and the client / master node - both ways !!