8

I am unable to read a file from HDFS using Java:

String hdfsUrl = "hdfs://<ip>:<port>";
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", hdfsUrl);
FileSystem fs = FileSystem.get(configuration);
Path filePath = new Path(hdfsUrl + "/projects/harmonizome/data/achilles/attribute_list_entries.txt.gz");
FSDataInputStream fsDataInputStream = fs.open(filePath);

SEVERE: Servlet.service() for servlet [edu.mssm.pharm.maayanlab.Harmonizome.api.DownloadAPI] in context with path [/Harmonizome] threw exception
java.lang.IllegalArgumentException: Wrong FS: hdfs://146.203.54.165:8020/projects/harmonizome/data/achilles/attribute_list_entries.txt.gz, expected: file:///
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
    at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:357)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
    at edu.mssm.pharm.maayanlab.Harmonizome.api.DownloadAPI.readLines(DownloadAPI.java:37)
    at edu.mssm.pharm.maayanlab.Harmonizome.api.DownloadAPI.doGet(DownloadAPI.java:27)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
...

I didn't setup our HDFS, so I don't know what I don't know. Any help is appreciated.

jds
  • 7,910
  • 11
  • 63
  • 101

3 Answers3

15

Try this:

Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(new URI(<url:port>), configuration);
Path filePath = new Path(<path/to/file>);
FSDataInputStream fsDataInputStream = fs.open(filePath);
BufferedReader br = new BufferedReader(new InputStreamReader(fsDataInputStream));

Please refer to http://techidiocy.com/java-lang-illegalargumentexception-wrong-fs-expected-file/

A similar problem is addressed.

jds
  • 7,910
  • 11
  • 63
  • 101
java_bee
  • 443
  • 4
  • 5
  • 7
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – scoa Aug 18 '15 at 17:49
  • I agree with @scoa. I was able to resolve my problem using this link and have edited your answer with the working code. – jds Aug 18 '15 at 18:06
  • @scoa thanks for suggestion. I will follow this in future. – java_bee Aug 19 '15 at 03:09
  • 2
    I don't get it. What was the root cause? – Worse_Username Aug 15 '17 at 13:45
  • Now the link is invalid & I can't know the root cause anymore – akki Jun 23 '21 at 09:12
  • 1
    Archived version: https://web.archive.org/web/20150805043209/http://techidiocy.com/java-lang-illegalargumentexception-wrong-fs-expected-file/ – theannouncer Jul 16 '21 at 18:17
  • From link, for posterity: "Your Configuration object is not initialized properly because of that it is trying to find this file in your local file system where it doesn’t exists because file [you are accessing] exists on HDFS." – theannouncer Jul 16 '21 at 18:21
2

Also below Configuration works!

Configuration conf = new Configuration();
conf.addResource(new Path("/usr/hdp/2.6.3.0-235/hadoop/etc/hadoop/core-site.xml"));
FileSystem fs = FileSystem.get(conf);
Sandeep Khot
  • 301
  • 3
  • 5
-1

I fixed this issue by changing the order of the class path. MapR uses their own libraries and somehow it was picking up the classes from another library.

MikeKulls
  • 873
  • 1
  • 10
  • 22