How to Append a File using Web HDFS REST API in C#?

Question

I am using Azure Data Lake to upload the file before but still want to append the text file content existing Data Lake text file. Is there any option available to append the text file data using Web HDFS REST API in C#?.

I am refer this link enter link description here

Code: I Can refer the above link get the append URL. But how can i use this URL and Append a File using c#?

private const string AppendUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?&op=APPEND&noredirect=true";

score 2 · Accepted Answer · answered Dec 28 '17 at 05:22

If you want to use the Rest Api to do that we could use the following code. I test it with Postman.

private const string AppendUrl = "https://{datalakeName}.azuredatalakestore.net/webhdfs/v1/{filepath}?append=true&op=APPEND&api-version=2016-11-01"
var token = "eyJ0eX.....";
using (var client = new HttpClient())
{
     client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
     var result = client.GetAsync(url).Result;
     var data = result.Content.ReadAsStringAsync().Result;
}

We also could use the Azure Microsoft.Azure.Management.DataLake.Store to do that. How to get the application id and secretkey you could refer to official document. More detail steps to get the permission to access the datalake you could refer to another SO thread.

var applicationId = "application Id";
var secretKey = "secretKey";
var tenantId = "tenant id";
var adlsAccountName = "datalake account name";
var creds = ApplicationTokenProvider.LoginSilentAsync(tenantId, applicationId, secretKey).Result;
var adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds,clientTimeoutInMinutes:60);
var stream = File.OpenRead(@"C:\tom\testtext.txt");
var test = adlsFileSystemClient.FileSystem.AppendWithHttpMessagesAsync(adlsAccountName, "test/abc.txt", stream).Result;

packages:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.Management.DataLake.Store" version="2.3.0-preview" targetFramework="net452" />
  <package id="Microsoft.Azure.Management.DataLake.StoreUploader" version="1.0.0-preview" targetFramework="net452" />
  <package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.13.8" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime" version="2.3.9" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime.Azure" version="3.3.9" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime.Azure.Authentication" version="2.2.0-preview" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="9.0.2-beta1" targetFramework="net452" />
</packages>

The Rest Api Code Works for me. Thanks you very much@Tom Sun — Saravanakumar Natarajan, Dec 28 '17 at 07:37

How to Append a File using Web HDFS REST API in C#?

1 Answers1