Some errors happen when loading data to HDFS

Question

I have a Java program trying to load data to HDFS:

public class CopyFileToHDFS {
   public static void main(String[] args) {
   try{
         Configuration configuration = new Configuration();

         String msg = "message1";
         String file = "hdfs://localhost:8020/user/user1/input.txt";
         FileSystem hdfs = FileSystem.get(new URI(file), configuration);
         FSDataOutputStream outputStream = hdfs.create(new Path(file), true);
         outputStream.write(msg.getBytes());
      }
      catch(Exception e){
        System.out.println(e.getMessage());
     }
 }
}

When I run the program, it gives me an error:

    java.util.ServiceConfigurationError:    org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3.S3FileSystem not found

It looks like some configuration issues. Can anyone give me some suggestions?

Thanks

I think you should use "hdfs:///" instead of "hdfs://" – Tanveer Dayan Apr 07 '16 at 05:30 — Tanveer Dayan, Apr 07 '16 at 05:30

score 0 · Answer 1 · edited May 23 '17 at 11:45

Something is specifying that org.apache.hadoop.fs.FileSystem includes S3. One possible cause is an old, stale META-INF file; see this Spark bug report.

If you're creating an uber-jar, it could be somewhere in there. If you can't find and eliminate the spec that's causing the problem, a work-around is to include AWS & Hadoop jars where the Spark driver/executors can find them; see this Stackoverflow question.

Some errors happen when loading data to HDFS

1 Answers1