6

I have such a nightmare with the Hadoop's Eclipse plugin. First of all the latest Hadoop version (which is 1.1.1) doesn't even include the Eclipse plugin jar file. version 1.0.4 has the plugin, but it doesn't work. Earlier versions of Hadoop such as 0.22.0 have the plugin and it works, but then these versions of Hadoop are old, and I want to work wit the latest version. So, what is the deal with Eclipse plugin for Hadoop? Why latest versions either don't have the plugin, or if they do, then the plugin doesn't work? Does everyone has so much problem with this plugin? Should I give up on this plugin?

Thanks, Shannon

user1888243
  • 2,591
  • 9
  • 32
  • 44
  • possible duplicate of [Where can I find the eclipse plugin for hadoop 1.0.4](http://stackoverflow.com/questions/13012852/where-can-i-find-the-eclipse-plugin-for-hadoop-1-0-4) – Chris Gerken Dec 09 '12 at 01:43

2 Answers2

5

I installed eclipse plugin for hadoop-1.0.4 yesterday with difficulty, and tested it successful.

The reason for plugin not working is that the jar package lost some libs :

  • commons-cli-1.2.jar
  • commons-configuration-1.6.jar
  • jackson-core-asl-1.8.8.jar
  • jackson-mapper-asl-1.8.8.jar
  • commons-httpclient-3.0.1.jar
  • commons-lang-2.4.jar

you could cp this jars from ${hadoop}/lib to ${jar}/lib, and don't forget modifying MANIFEST.

For convenience, I add some code to ${eclipse-plugin-src}/build.xml target jar

<copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-configuration-${commons-configuration.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-httpclient-${commons-httpclient.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-lang-${commons-lang.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-core-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-mapper-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>

and modified MANIFEST.MF in ${eclipse-plugin-src}/META-INF

undle-ClassPath: classes/,
 lib/hadoop-core.jar,
 lib/commons-cli-1.2.jar,
 lib/commons-configuration-1.6.jar,
 lib/jackson-core-asl-1.8.8.jar,
 lib/commons-httpclient-3.0.1.jar,
 lib/jackson-mapper-asl-1.8.8.jar,
 lib/commons-lang-2.4.jar

rebuild eclipse-plugin, and HAVE FUN!

hs3180
  • 188
  • 1
  • 8
  • Thanks Chris. You are a life saver. Just one other question; I took your earlier advice and started creating Hadoop projects through maven which means I won't need Eclipse plugin for such projects. Do you think I should just stop using Eclipse plugin? I think the difficulties that come up with Hadoop's Eclipse plugin makes it somewhat of a liability. Do you agree? – user1888243 Dec 11 '12 at 23:35
  • Hadoop's Eclipse plugin supply eclipse debug support for hadoop map/reduce and hdfs application, give a GUI to connect hdfs and jobtracker – hs3180 Dec 16 '12 at 22:58
  • I have followed your methods and built the plugin with command "ant jar", but I got noting. Can you express your method in more detail? Thanks! @hs3180 – jerry_sjtu Dec 24 '12 at 05:30
  • @jerry_sjtu can you put some build info ? ( stdout / stderr result of ant) you'd better build jar using build.xml in ${plugin_src}. – hs3180 Dec 25 '12 at 15:31
  • I have built the plugin when providing parameter '-Doverride' and 'eclipse.home', but it seems that the plugin built doesn't work well with my eclipse 'INDIGO' in ubuntu. – jerry_sjtu Dec 26 '12 at 06:28
  • @hs3180: I am using this command: ant clean package -Djava5.home=/opt/java/jdk1.5.0_22 -Dforrest.home=/opt/apache-forrest-0.8 Using eclipse galileo and put eclipse.home in hadoop/src/contrib/eclipse-plugin/build.properties but still error, its because of "autoreconf" so I installed it still build failed... Even I did all the configuration you specified it is giving me compiler error while running following command: /src/contib/eclipse-plugin>ant jar Is it necessary that hadoop cluster is running for creating eclipse plugin? – Chirag Tayal Feb 20 '13 at 20:51
  • @ChiragTayal: It doesn't need hadoop cluster running. Only need a hadoop-core jar – hs3180 Feb 21 '13 at 19:36
  • @hs3180 Thanks so much for the help. I get strange errors - {plugin_src_dir}/eclipse/HadoopPerspectiveFactory.java:22: package org.eclipse.jdt.ui does not exist. Any advice? (Attempting to build for Indigo) – Eitan Mar 04 '13 at 21:37
2

Well, combined with hs3180's answer, here is what I actually did in configuring the compiling.

  1. In ${eclipse-plugin-src}/build.xml, use the following directives to include required jars by the eclipse-plugin. Note here that, instead of from ${hadoop.root}/build, copying of the hadoop-core-${version}.jar is now from ${hadoop.root} directly, because if you are using the compiled version of hadoop, the ${hadoop.root}/build folder would be actually empty. Copying of the commons-cli-${commons-cli.version}.jar is now from ${hadoop.root}/lib for the same reason. 

    <!-- <copy file="${hadoop.root}/build/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/> -->
<!-- <copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/> -->
<copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-configuration-${commons-configuration.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-httpclient-${commons-httpclient.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-lang-${commons-lang.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-core-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-mapper-asl-${jackson.version}.jar"  todir="${build.dir}/lib" verbose="true"/>

  2. Modify the ${eclipse-plugin-src}/META-INF/MANIFEST.MF file to accommodate the changes made to ${eclipse-plugin-src}/build.xml.

    Bundle-ClassPath: classes/,
 lib/hadoop-core.jar,
 lib/commons-cli-1.2.jar,
 lib/commons-configuration-1.6.jar,
 lib/jackson-core-asl-1.8.8.jar,
 lib/commons-httpclient-3.0.1.jar,
 lib/jackson-mapper-asl-1.8.8.jar,
 lib/commons-lang-2.4.jar

  3. Modify the ${hadoop.root}/src/contrib/build-contrib.xml file at two places. The first is to set up the properties needed for the plugin compiling. And the second is to make sure jars, especially hadoop-core-1.0.4.jar, in the ${hadoop.root} folder are visible to javac, because the eclipse-plugin references the hadoop classes. This setup is different from the first two in purpose and thus not dispensable.

    <!-- Properties added for compiling eclipse-plugin -->
<!-- http://yiyujia.blogspot.com/2012/11/build-hadoop-eclipse-plugin-from-source.html -->
<property name="eclipse.home" location="/Users/xuj/Downloads/eclipse/"/>
<property name="version" value="1.0.4"/>
<property name="commons-cli.version" value="1.2"/>

<!-- the normal classpath -->
<path id="contrib-classpath">
  <fileset dir="${hadoop.root}">
    <include name="*.jar" />
  </fileset>
  <!-- more path elements go here -->
</path>

When all files correctly configured, call 'ant jar' from ${eclipse-plugin-src}/ in the console shall be sufficient for the rest.

lcn
  • 2,239
  • 25
  • 41