I am trying to read Parquet file which is located in Azure DataLake Gen2 Container for automation framework. Able to connect using StorageAccount & AccessKey with information provided above URL. However I want to temporary read it to DataTable / SQL View / any other option which give me option to query and Assert records while the system is in test.
Getting System.IO. cannot convert to azure.storage.files.modesl error exception. Getting 200 response.
Unable to convert Parquet data from filestream.
Using NUnit 2019 Community Edition (c#).
DataLakeFileClient fileClient = fileSystemClient.GetFileClient(fileName);
Response<FileDownloadInfo> downloadResponse = fileClient.Read();
var reader = new StreamReader(downloadResponse.Value.Content);
Stream fileStream = GetSteamfromString(lines);
public static Stream GetSteamfromString(string lines)
{
var stream = new MemoryStream();
var writer = new StreamWriter(stream);
writer.Write(lines);
writer.Flush();
stream.Position = 0;
return stream;
}
Note - Test is to compare Raw Data (Blob Storage) to Cleaned data (DataLakeGen2). Storing data from blob into DataTable to assert the conditions/tests.
Thanks,