0

I am trying to test if I can jar 3rd party dependencies into my jar, SSH the jar over to a remote machine, and then run a map reduce job.

Process:

With my project, I run mvn clean package and that produces the files my-appy-1.0-SNAPSHOT.jar and original-my-appy-1.0-SNAPSHOT.jar. I scp the first file over to my remote machine and run the command:

hadoop jar hadoop my-appy-1.0-SNAPSHOT.jar  /user/bli1/wordcount/input /user/bli1/wordcount/output

I also tried:

hadoop my-appy-1.0-SNAPSHOT.jar WordCount /user/bli1/wordcount/input /user/bli1/wordcount/output

I'm not sure why I am getting this error:

Error: Could not find or load main class my-appy-1.0-SNAPSHOT.jar

enter image description here

pom.xml:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.18.1</version>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.1</version>
        <configuration>
          <source>1.7</source>
          <target>1.7</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.2</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <createDependencyReducedPom>false</createDependencyReducedPom>
              <artifactSet>
                <excludes>
                  <exclude>org.hamcrest:*</exclude>
                  <exclude>org.mockito:*</exclude>
                  <exclude>org.objenesis:*</exclude>
                </excludes>
              </artifactSet>
              <filters>
                <filter>
                  <artifact>*:*</artifact>
                  <excludes>
                    <exclude>META-INF/LICENSE</exclude>
                    <exclude>META-INF/license</exclude>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                  </excludes>
                </filter>
              </filters>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <manifestEntries>
                    <Main-Class>com.mycompany.app.WordCount</Main-Class>
                    <Build-Number>1</Build-Number>
                  </manifestEntries>
                </transformer>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
Tunaki
  • 132,869
  • 46
  • 340
  • 423
Liondancer
  • 15,721
  • 51
  • 149
  • 255

3 Answers3

1

Open your *.jar file and check it's MANIFEST.mf file. It should contain the "Main-Class" row with your class with "public static main()" method. If there is no "Main-Class" add it yourself

This link will help you do this: https://docs.oracle.com/javase/tutorial/deployment/jar/appman.html

kerradus
  • 36
  • 4
  • I am using the shade plugin and I think all I need to do is declare it in my pom.xml:` com.mycompany.app.WordCount`. Please correct me if I am wrong – Liondancer Mar 29 '15 at 23:29
1

you need to give full path to main class including package name.

hadoop my-appy-1.0-SNAPSHOT.jar com.mycompany.app.WordCount /user/bli1/wordcount/input /user/bli1/wordcount/output
Karthik
  • 1,801
  • 1
  • 13
  • 21
0

The command you run through jar file is as below: hadoop jar jar_filename.jar package_name.class_name HDFS_inputfile_name HDFS_output_directory

Your command should be

hadoop jar my-appy-1.0-SNAPSHOT.jar com.mycompany.app.WordCount /user/bli1/wordcount/input /user/bli1/wordcount/output

Nandakishore
  • 981
  • 1
  • 9
  • 22