0

I am logging some data to a Gen1 Azure Datalake Store, using the Microsoft.Azure.DataLake.Store driver.

I am authenticating and creating a client like so:

var adlCreds = await ApplicationTokenProvider.LoginSilentAsync(tenant, clientId, secret);
var adlClient = AdlsClient.CreateClient(dlUrl, adlCreds);

And then writing to a file using ConcurrentAppendAsync like so (file name just an example):

var textBytes = Encoding.UTF8.GetBytes(appendText);
await client.ConcurrentAppendAsync("test/myFile.json", true, textBytes, 0, textBytes.Length);

This is working, most of the time, however I am seeing intermittent errors logged with the error:

CONCURRENTAPPEND failed with Unknown Error: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

When this occurs the concurrent append fails and the data is not saved.

The service this is running in is on .NET Framework 4.6.1. I am caching and reusing the credentials object (for 30 mins) and the AdlsClient (for 5 mins) which as far as I can tell is OK, so I'm not sure what the issue is.

Pavel Anikhouski
  • 21,776
  • 12
  • 51
  • 66
QTom
  • 1,441
  • 1
  • 13
  • 29
  • What do mean " I am caching and reusing the credentials object (for 30 mins) and the AdlsClient (for 5 mins)"? – Jim Xu Sep 17 '19 at 01:44

1 Answers1

0

One of the problems could be due to the use of "Create" and "ConcurrentAppend" on the same file stream. ADLS documentation mentions that they can't be used on the same file. If you are using them concurrently, maybe, try changing the "Create" command to "ConcurrentAppend" as the latter can be used to create a file if it doesn't exist.

chiragMishra-msft
  • 192
  • 2
  • 4
  • 30
  • I am not using the Create command, I am letting ConcurrentAppend create the file as needed. The only command hitting my datalake is ConcurrentAppend – QTom Sep 17 '19 at 10:49