Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStrea

Question

I'm pretty new on Cloudera Quick-start so sorry if my explanation will be not so clear. Anyway I'm writing a code in Java which read File from Hdfs. I build a Maven-Project and I set up all the dependencies in the pom.xml, but when I try to launch the jar from shell (java -jar jnameofthefile.jar) I'm getting this error: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStrea

This is my Java code:

package com.hdfs_java_api;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;



import java.io.InputStream;
import java.io.IOException;
import java.net.URI;

public class HadoopFileSystemCat {

public static void main(String [] args) throws IOException
{
    String uri = "hdfs://quickstart.cloudera:8020/user/hive/warehouse/Orders.csv";
    Configuration conf = new Configuration();
    FileSystem fs = FileSystem.get(URI.create(uri), conf);
    InputStream in = null;

    try {

    in = fs.open(new Path(uri));


    IOUtils.copyBytes(in, System.out, 4096, false);
}finally{
    IOUtils.closeStream(in);
        }
}


}

And this is my pom.xml:

 <project xmlns="http://maven.apache.org/POM/4.0.0" 
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com</groupId>
<artifactId>cards</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>

<name>hdfs_java_api</name>
<url>http://maven.apache.org</url>

<properties>
<project.build.sourceEncoding>UTF- 
8</project.build.sourceEncoding>
</properties>

<dependencies>
<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>3.8.1</version>
  <scope>test</scope>
</dependency>

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-common</artifactId>
  <version>2.6.0-cdh5.13.0</version>
</dependency>

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-yarn-common</artifactId>
  <version>2.6.0-cdh5.13.0</version>
</dependency>

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-mapreduce-client-common</artifactId>
  <version>2.6.0-cdh5.13.0</version>
</dependency>

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-mapreduce-client-core</artifactId>
  <version>2.6.0-cdh5.13.0</version>
</dependency>
</dependencies>

<repositories>
<repository>
  <id>cloudera</id>
  <name>cloudera</name>
  <url>https://repository.cloudera.com/artifactory/cloudera- 
   repos/</url>
</repository>
</repositories>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
                <archive>
                    <manifest>

  <mainClass>com.hdfs_java_api.HadoopFileSystemCat</mainClass>
                    </manifest>
                </archive>
            </configuration>
        </plugin>
    </plugins>
</build>

As i said I'm noob so be patient and try to be as much clear as possible, thank you in advance!

Ninad Pingale · Answer 1 · 2018-09-28T13:33:10.023

1

I think you are missing core library

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>0.20.2</version>
</dependency>

After that make sure, you have included "Maven Dependencies" in build path.

And in Deployment Assembly -

edited Sep 28 '18 at 13:33

answered Sep 28 '18 at 11:07

Ninad Pingale

6,801
5
32
55

I have added the core library but the same error come out – PdF Sep 28 '18 at 12:22
Is it properly downloaded ? Use maven>update project. – Ninad Pingale Sep 28 '18 at 13:04
I did it, nothing change – PdF Sep 28 '18 at 13:23
First of all thx for your time, I included Maven dependencies but for the second step the Deployment Assembly was missing. So I added org.eclipse.wst.common.modulecore.ModuleCoreNature to my .project file as explained here [http://stackoverflow.com/questions/8399885/eclipse-deployment-assembly-does-not-appear-in-properties] and it appeared but there was an error which said that this is not a virtual project, after some time Eclipse crashed. Maybe I need to reinstall eclipse also because it is photon version not EE. – PdF Sep 28 '18 at 14:39
I think you should follow other things also provided in this link by other users http://stackoverflow.com/questions/8399885/eclipse-deployment-assembly-does-not-appear-in-properties – Ninad Pingale Sep 30 '18 at 14:20

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStrea

1 Answers1