Directory size in Azure Data Lake Store

Question

Is there a simple API using which I can get the size of an ADLS directory? preferably something in C#, but it's not a must.

score 3 · Answer 1 · answered Dec 01 '17 at 08:52

We could use the Get Content Summary of a Directory REST API to do that.

curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETCONTENTSUMMARY"

C# code demo

 var url = "https://tomdatalake.azuredatalakestore.net/webhdfs/v1/tomtest?api-version=2017-08-01&op=GETCONTENTSUMMARY";
 var token = "eyJ0eX.....";
 using (var client = new HttpClient())
 {
     client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
     var result = client.GetAsync(url).Result;
     var data = result.Content.ReadAsStringAsync().Result;
 }

I also test it with PostMan.

score 2 · Accepted Answer · answered Dec 01 '17 at 00:43

Yes, you can use DataLakeStoreFileSystemManagementClient.FileSystem.GetContentSummary:

var client = new DataLakeStoreFileSystemManagementClient(credentials);
ContentSummaryResult result = client.FileSystem.GetContentSummary(dataLakeAccount, path);
var dirSize = result.ContentSummary.Length;

Documentation.

Directory size in Azure Data Lake Store

2 Answers2