1

I have a simple java code to copy a text file from my local to the hdfs. I am using cloudera's quickstart virtual machine. 

Configuration conf = new Configuration();
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("/home/cloudera/workspace/Downloader/output/data.txt"), 
                          new Path("hdfs://quickstart.cloudera:8020/user/cloudera/"));

I get this error after running this code:

Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://quickstart.cloudera:8020/user/cloudera, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
at org.apache.hadoop.fs.LocalFileSystem.pathToFile(LocalFileSystem.java:61)
at org.apache.hadoop.fs.LocalFileSystem.exists(LocalFileSystem.java:51)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:355)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:211)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:67)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1143)

What could I be doing wrong?

Free Man
  • 195
  • 3
  • 13

2 Answers2

1

I resolved this problem. You have to be careful with the kind of jar files you add to your classpath especially when working with cloudera quickstart vm. If available, use jar files provided by cloudera. They can be found in this folder: /usr/lib/hadoop/client/ This code will work fine without any problems.

Configuration conf = new Configuration();
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("/home/cloudera/workspace/Downloader/output/data.txt"), 
                      new Path("hdfs://quickstart.cloudera:8020/user/cloudera/"));
Free Man
  • 195
  • 3
  • 13
0

Not correct path("hdfs://quickstart.cloudera:8020/user/cloudera/" use this example:

Configuration conf = getConf();
System.out.println("fs.default.name : - " + conf.get("fs.default.name"));
// It prints uri  as : hdfs://10.214.15.165:9000 or something...
String uri = conf.get("fs.default.name");
FileSystem fs = FileSystem.get(uri,getConf());
Fortran
  • 2,218
  • 2
  • 27
  • 33
  • when i run your code it says the default name is : "file:///" – Free Man Oct 21 '15 at 19:50
  • I replaced "hdfs://quickstart.cloudera:8020/user/cloudera/" with "file:///user/cloudera/... Then I got a new error: "Mkdirs failed to create "file/user – Free Man Oct 21 '15 at 19:55
  • interestingly, when I changed your line "String uri = conf.get(fs.default.name") to "String uri = get ("fs.defaultFS") .... the default name was registered as hdfs://quickstart.cloudera:8020 – Free Man Oct 21 '15 at 20:41