Read a hadoop config Files from Unix Path in Java Program

Question

I am developing an Java Application and this application is saving a result data to HDFS. The java Application should run in my windows machine.

As of now, I copied the Hadoop config files such as core-site, hdfs-site.xml in to my windows machine and testing it. Below is my configuration Code Snippet.

config.addResource(new Path("C:\\Hadoop\\core-site.xml"));
config.addResource(new Path("C:\\Hadoop\\hdfs-site.xml"));
config.addResource(new Path("C:\\Hadoop\\mapred-site.xml"));
config.addResource(new Path("C:\\Hadoop\\yarn-site.xml"));

Is there any way I can load config files directly from unix server from my Windows Machine like below.

config.addResource(new Path("/opt/hdp/2.3.4.7-4/hadoop/conf/core-site.xml"));
config.addResource(new Path("/opt/hdp/2.3.4.7-4/hadoop/conf/hdfs-site.xml"));
config.addResource(new Path("/opt/hdp/2.3.4.7-4/hadoop/conf/mapred-site.xml"));
config.addResource(new Path("/opt/hdp/2.3.4.7-4/hadoop/conf/yarn-site.xml"));

Kindly help me.

Does your Windows machine have the Unix server's disk mounted (via Samba, for example)? If not, the answer is no. — Jim Garrison, Jul 05 '16 at 06:42
No..There is no unix server disk mounted in my local windows machine. — Padmanabhan Vijendran, Jul 05 '16 at 06:43
Then it's not clear what you are asking. There's no magic trick that will read the files from the Unix server on Windows. — Jim Garrison, Jul 05 '16 at 06:45
I am not asking for a magic trick. I am asking Is there any API which I can import to establish the connection to the unix server. — Padmanabhan Vijendran, Jul 05 '16 at 07:00

score 1 · Accepted Answer · edited May 23 '17 at 12:09

You could theoretically accomplish this with some work. There are two possibilities, SAMBA or NFS. In both cases, the Unix server would have to be running the server component and be configured to export the desired filesystem, and you would need authentication credentials that grant you access to the exported filesystem.

For SAMBA you might be able to use the suggestions in this question to map the network drive. You'll have to write code to detect if the drive is already mapped, and cope with assigning a drive letter.

For NFS you will need to find an NFS client library for Java.

I suppose a third possibility is to copy the files from the Unix server via FTP or (better) SCP.

score 0 · Answer 2 · answered Jul 05 '16 at 08:50

It looks like you are trying to embed a hadoop/hdfs instance. Most hdfs instances are likely to be multibox perhaps in a data centre. If you include the appropriate jars, and setup permissions you should be able to interact with hdfs via the namenode using urls:

hdfs://namenode:port/path/to/your/file

If it is for testing then spinning up a local instance independent from the prod cluster makes sense but it probably needs it's own independent config.

Read a hadoop config Files from Unix Path in Java Program

2 Answers2