0

I'm trying to connect to remote hive from within my spark program in Intellij installed on local machine.

I placed the hadoop cluster config files on local machine and configured environment variables HADOOP_CONF_DIR in Intellij run configurations of this spark program to be able to detect this hadoop cluster but intelliJ is somehow not reading these files and spark program defaults to local hive metastore instance.

Is there anyway to configure intelliJ to read hadoop config files locally. Any help is highly appreciated.

hitesh sahni
  • 1
  • 1
  • 3

2 Answers2

0

Please configure SPARK_CONF_DIR variable and copy the hive-site.xml in that directory. Spark will connect the specified hive meta-store and make sure that hive-site.xml points to your cluster details.

Thanks Ravi

Ravikumar
  • 1,121
  • 1
  • 12
  • 23
  • Hi Ravi,Thanks for your response. I already tried that - copied hive-site.xml and hdfs-site.xml on the local directory and pointed SPARK_CONF_DIR to that path but it doesnot work. – hitesh sahni Feb 14 '18 at 20:48
  • Hi Hitesh, could you please check are you able to access remote hive from your local machine by running the hive cli command **hive --config /etc/spark/conf** replace the /etc/spark/conf to SPARK_CONF_DIR path. – Ravikumar Feb 14 '18 at 22:43
  • Hi Ravi, I'm able to connect to Hive from local spark setup via spark-shell but problem is only through IDE (IntelliJ), it is not able to read the config files. Were you able to have IntelliJ or other IDE connect to remote hive through these config files. – hitesh sahni Feb 15 '18 at 09:02
  • Yes, i was able to resolve this. Please add required hadoop conf (xml) files to the resources directory of your project and then spark will be able to pick up the hive configuration to connect automatically. – hitesh sahni Sep 07 '18 at 09:42
0

Add hadoop configuration files folder to intellij project class path

Project Settings -> Libraries -> + -> java -> select folder with all the config files -> classes

spats
  • 805
  • 1
  • 10
  • 12