9

I am trying to run Spark using yarn and I am running into this error:

Exception in thread "main" java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.

I am not sure where the "environment" is (what specific file?). I tried using:

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

in the bash_profile, but this doesn't seem to help.

halfer
  • 19,824
  • 17
  • 99
  • 186
user1072337
  • 12,615
  • 37
  • 116
  • 195

3 Answers3

16

While running spark using Yarn, you need to add following line in to spark-env.sh

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

Note: check $HADOOP_HOME/etc/hadoop is correct one in your environment. And spark-env.sh contains export of HADOOP_HOME as well.

Shubhangi
  • 2,229
  • 2
  • 14
  • 14
0

For the Windows environment, open file load-spark-env.cmd in the Spark bin folder and add the following line:

set HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop
player0
  • 124,011
  • 12
  • 67
  • 124
Sidharth Ramalingam
  • 1
-2

just an update to answer by Shubhangi,

 cd $SPARK_HOME/bin
 sudo nano load-spark-env.sh

add below lines , save and exit

export SPARK_LOCAL_IP="127.0.0.1"

export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"

export YARN_CONF_DIR="$HADOOP_HOME/etc/hadoop"

Raghavendra J.P
  • 1