How to Authenticate WebHDFS with c#

Question

I have been attempting to upload files using c# into Hadoop using the WebHDFS REST API. This code works fine:


    using (var client = new System.Net.WebClient())
    {
       string result = client.DownloadString("http:/ /host:50070/webhdfs/v1/user/myuser/?op=LISTSTATUS");
       client.DownloadFile("http:/ /host:50070/webhdfs/v1/user/myuser/tbible.txt?user.name=myuser&op=OPEN","d:\tbible.txt");
    }

This code gets a 403 Forbidden:


    using (var client = new System.Net.WebClient())
    {

       client.UploadFile("http:/ /host:50070/webhdfs/v1/user/myuser/?user.name=myuser&op=CREATE", "PUT", "d:\bible.txt");
    }

I have tried adding a network credential, with no luck. How do I authenticate to our Cluster from .NET? The Cluster is Hortonworks HDP1.3 on RHEL5. (The extra spaces in this post are to keep http:/ / from being a link) Also, I would have liked to use Microsoft's hadoop SDK, but it is alpha and wont compile in my environment :(

I got it to work. I used fiddler to diagnose the http stream. I just added the target file name to fix the problem: client.UploadFile("http://host:50070/webhdfs/v1/user/myuser/bible.txt?user.name=myuser&op=CREATE", "PUT", "d:\\bible.txt"); I still dont know how to authenticate. — user2732948, Aug 30 '13 at 19:57

score 0 · Answer 1 · answered Oct 26 '13 at 17:10

Make sure that you are writing to a directory that is under the group which WebHDFS operates under. By default this is hdfs.

A quick way to check this doing hadoop fs -ls on the directory's parent directory to get the group permission settings (the second column that may look like a username).

How to Authenticate WebHDFS with c#

1 Answers1

Linked