2

I'm trying to run the spark application on standalone mode, after installing the spark when i tried to run spark-submit command found the above error no java file found. Tried two different approaches.

  • approach-1 : able to remove the additional '/' by changing the environment file but still the issue persists
  • approach-2 : made the files having java home consistent but unable to find the spark.conf file where I could see to make it consistent.
  • approach-3 : Tried to change the bash profile but no result either

Below is my bash_profile

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home

export SPARK_HOME=/Users/xxxx/server/spark-2.3.0-bin-hadoop2.7

export SBT_HOME=/Users/xxxx/server/sbt

export SCALA_HOME=/Users/xxxx/server/scala-2.11.12

export PATH=$JAVA_HOME/bin:$SBT_HOME/bin:$SBT_HOME/lib:$SCALA_HOME/bin:$SCALA_HOME/lib:$PATH

export PATH=$JAVA_HOME/bin:$SPARK_HOME:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

export PYSPARK_PYTHON=python3

Here is my etc/environment file:

JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home"

Could any help me in resolving this issue trying to run spark-scala application from 2 days on my Mac.

InstanceDeveloper
  • 89
  • 11

3 Answers3

0

This should help you How to get java path in CentOS?

They usually reside in /usr/lib/jvm. You can list them via ll /usr/lib/jvm. The value you need to enter in the field JAVA_HOME in jenkins is /usr/lib/jvm/jdk1.8.0_192

Here bin path should be present put the as Java Home path.

tarun
  • 218
  • 2
  • 11
0

First, you can check that the Java environment variable was configured successfully, and then check that the java_home path in "sparkenv.conf" exists and that it is correct. I hope I can help you.

keven
  • 46
  • 1
  • I couldn't find the `sparkenv.conf` file in my mac instead i found `spark-env.sh` in `/Users/xxxx/server/spark-2.3.0-bin-hadoop2.7/conf` and i added java_home to it and yet i could see the same issue. – InstanceDeveloper Dec 05 '19 at 00:32
  • That's right, the "spark env.conf" file. You can view the path to your Java using the command "which Java" on the MAC terminal. Then "java -version" checks to see if the version of Java is compatible with saprk. – keven Dec 05 '19 at 01:36
0

In case you are running in Python, you can set Java environmental variable as follows:

Step 1. Find out where your Java/jdk is:

echo $(/usr/libexec/java_home -v 1.8)

(or ! echo $(/usr/libexec/java_home -v 1.8) in Jupyter notebook cell)

Step 2. Manually configure Java location:

import os
os.environ["JAVA_HOME"] = <path-you-obtained>

This worked for me for using pyspark 2.4.x

David C.
  • 1,974
  • 2
  • 19
  • 29