No FileSystem for scheme: hdfs when building fat jar in Spark; works fine in Eclipse mars

Question

I get that error when I do a fat jar with my spark program and then I launch it using:

java -jar -myApp.jar [args]

When I run my app in Eclipse, it works perfectly well. Here is my POM:

 <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
   <maven.compiler.source>1.7</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
  </properties>

  <dependencies>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka_2.10</artifactId>
    <version>1.5.0</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.10</artifactId>
    <version>0.9.0.1</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.10</artifactId>
    <version>1.6.1</version>
</dependency>
<dependency>
    <groupId>com.google.code.gson</groupId>
    <artifactId>gson</artifactId>
    <version>2.6.2</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.6.0</version>
</dependency>
 </dependencies>
 <build>
<plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.1</version>
                <executions>
                    <execution>
                    <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <source>1.7</source>
                            <target>1.7</target>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>reference.conf</resource>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>gft.hbaseMedia.MainSparkCopyHDFS</mainClass>
                                </transformer>
                            </transformers>
                                <filters>
                                    <filter>
                                        <artifact>*:*</artifact>
                                        <excludes>  
                                            <exclude>META-INF/*.SF</exclude>
                                            <exclude>META-INF/*.DSA</exclude>
                                            <exclude>META-INF/*.RSA</exclude>
                                        </excludes>
                                    </filter>
                                </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
    </plugins>
    <resources>
            <resource>
                <directory>resources</directory>
            </resource>
        </resources>
  </build>
</project>

The piece of code that gives problems is:

Configuration conf = new Configuration();
conf.set("fs.defaultFS", hdfsConfPath);
FileSystem fs = FileSystem.get(conf);

I know it's Maven that's screwing up things somehow, but I don't know how to solve this issue!

Refer http://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file — Amit Mahajan, Jul 12 '16 at 22:07
@amitmah I had already tried and I simply get a different error. About a deprecated class missing — Vale, Jul 12 '16 at 22:20
have you checked the maven configuration of eclipse and point to right settings.xml file. Maybee your eclipse embedded maven version are different. — Amit Mahajan, Jul 13 '16 at 01:33

score 1 · Accepted Answer · answered Jul 13 '16 at 14:28

It looks like the answer @amitmah pointed me to was indeed the right one (talking about the second most voted one, the maven-shade transformer).
The first one gives a ClassNotFound Exception and the maven shade transformer one gives it too.
The solution for me was to use only what Sparks already has,
Apply the answer given in the comments under my question (for people who use Maven Shade),
delete the hdfs dependency, as it creates some kind of conflicts at runtime, so no error during the maven build.
Here are the transformers in the pom:

 <transformers>
         <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
         <resource>reference.conf</resource>
         </transformer>
         <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
         <mainClass>gft.hbaseMedia.MainSparkCopyHDFS</mainClass>
         </transformer>
         <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>

No FileSystem for scheme: hdfs when building fat jar in Spark; works fine in Eclipse mars

1 Answers1