Installed hadoop using brew, now want to run hadoop jobs in intelliJ IDEA. How to setup the environment and resolve dependencies?
Asked
Active
Viewed 6,087 times
1 Answers
3
After spending lot's of time, was finally able to run hadoop jobs from intelliJ IDEA. Hope this might save time for others.
brew upgrade hadoop
installed/updated hadoop in /usr/local/Cellar/hadoop/2.x.x Now, we will try to import dependency from this location in our intelliJ
- Launch IntelliJ and click on ‘Create New Project’
- Create a java project
- Go to File -> Project Structure -> Select Project Module in Modules -> Select Dependencies Tab -> Select ‘+’ in bottom and click Jars or Directories -> Import dependencies from Hadoop installation
- For importing dependencies, first select all folders inside ../hadoop- 2.x.x/share/hadoop/
Repeat Step 4 and import ../Hadoop-2.x.x/share/hadoop/common/lib folder
Hadoop environment is integrated with intelliJ Java Project. You can create new java classes to implement your drivers, mappers, reducers etc.
Atul Kaushik
- 5,181
- 3
- 29
- 36
-
1You should really use Maven/Gradle instead of adding the JAR files manually – OneCricketeer Nov 25 '17 at 17:48
-
Hi @cricket_007 , could you please provide step by step process do so. I am not an expert in Maven/Gradle based IDEs – Atul Kaushik Nov 25 '17 at 19:06
-
Too broad for Stackoverflow, sorry. Main point - you can use any or no IDE which is useful to build your code within an actual Hadoop cluster – OneCricketeer Nov 25 '17 at 23:47