Error writing a file to Azure Data Lake from an Azure function using MSI

Question

I am trying to create an Azure function that writes to Azure Data Lake Store. I am using Managed Service Identity to, well, manage the authentication stuff.

I have enabled MSI on the Function app. I have also enabled the Function app to access the required Data Lake Store. And I am using the following code to get a token and write to ADL. AM I missing something?

var azureServiceTokenProvider = new AzureServiceTokenProvider();
string accessToken = await azureServiceTokenProvider.GetAccessTokenAsync("https://datalake.azure.net");
var client = AdlsClient.CreateClient(_adlsAccountName, accessToken);
using (var stream = client.CreateFile(fileName, IfExists.Overwrite))
    {
        byte[] textByteArray = Encoding.UTF8.GetBytes("Winter is coming! \r\n");
        stream.Write(textByteArray, 0, textByteArray.Length);
    }

My code fails with the following error.

with exception Microsoft.Azure.DataLake.Store.AdlsException: Error in creating file /Path/tempFile0.txt.

**Operation: CREATE failed with HttpStatus:Unauthorized Error: Uexpected error in JSON parsing.**

Last encountered exception thrown after 1 tries. [Uexpected error in JSON parsing]

[ServerRequestId:<Some ID>]

at Microsoft.Azure.DataLake.Store.AdlsClient.CreateFile(String filename, IfExists mode, String octalPermission, Boolean createParent)

score 4 · Accepted Answer · answered Apr 24 '18 at 23:09

Adding "Bearer " to the access token worked for me. Like thus (everything else remains the same),

var client = AdlsClient.CreateClient(_adlsAccountName, “Bearer “ + accessToken);

Thanks in part to Arturo Lucatero's Github doc which mentioned this. https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/active-directory/managed-service-identity/tutorial-linux-vm-access-datalake.md

score 0 · Answer 2 · answered Apr 24 '18 at 06:59

I mainly use following snippet to authenticate and write to Data Lake from Azure Functions:

var clientCredential = new ClientCredential(clientId, clientSecret);
var creds = ApplicationTokenProvider.LoginSilentAsync("domainId", clientCredential).Result;
_client = new DataLakeStoreFileSystemManagementClient(creds);

Where clientId and clientSecret are:

an ObjectId from AD
a secret attached to it

So basically you have to create a service principal and obtain those particular properties from the portal.

Then I'm able to use following:

public async Task AppendToFile(string destinationPath, string content)
{
    using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(content)))
    {
        await _client.FileSystem.ConcurrentAppendAsync("datalakeaccount", destinationPath, stream, appendMode: AppendModeType.Autocreate);
    }
}

To write data to ADLS.

You can also refer to this blog post.

Thanks, @Kamo but I would like to use MSI to do the authentication so I can avoid storing the client ID and secret in a Key Vault. — MV23, Apr 24 '18 at 17:57

Error writing a file to Azure Data Lake from an Azure function using MSI

2 Answers2