1

I am trying to run spark-submit:

/usr/local/Cellar/apache-spark/2.3.0/libexec/bin/spark-submit  sdp-consumer.py

It gives error:

/usr/local/Cellar/apache-spark/2.3.0/libexec/bin/spark-submit: line 27: /usr/local/Cellar/apache-spark/2.2.0/libexec/bin/spark-class: No such file or directory

Why is it complaining about 2.2.0 when that does not exists? (I do not have any ENV that points to 2.2.0)

How to use spark-submit or how to solve this issue?

Thanks.

Joe
  • 11,983
  • 31
  • 109
  • 183
  • Just to confirm, your `$SPARK_HOME` is blank? You should be able to `cat` your `spark-submit` (it's a bash script) and see that it invokes `"${SPARK_HOME}"/bin/spark-class`. – chrisaycock Apr 18 '18 at 19:10
  • echo $SPARK_HOME `/usr/local/Cellar/apache-spark/2.3.0/libexec` So still not sure why is it going to 2.2.0. – Joe Apr 18 '18 at 19:15
  • What happens when you `source /usr/local/Cellar/apache-spark/2.3.0/libexec/bin/find-spark-home`? Does your `$SPARK_HOME` change then? – chrisaycock Apr 18 '18 at 19:35

0 Answers0