1

I would like to do something simple!!!

Copy a blob from a Container (SourceContainer) which is in a dataLake gen 2 (SourceDataLake) to a second DatLake (TargetDataLake) via c# code in an Azure Functions.

The connection between my Azure Function and SourceDataLake is secured via Private Link and Private Endpoint.

DataLakeDirectoryClient sourcedirectoryClient2 = sourceDataLakeFileSystemClient.GetDirectoryClient(myPath);
DataLakeFileClient sourcefileClient = sourcedirectoryClient2.GetFileClient(myBlobName);
Response<FileDownloadInfo> downloadResponse = await sourcefileClient.ReadAsync(); //I get the error in this line
Stream reader = downloadResponse.Value.Content;

DataLakeDirectoryClient targetdirectoryClient = taregetDataLakeFileSystemClient.GetDirectoryClient(TargetDirectory);
DataLakeFileClient targetfileClient = await targetdirectoryClient.CreateFileAsync(myBlobName);

await targetfileClient.UploadAsync(reader, true);

for Authentication to DataLake I use this function:

public static DataLakeFileSystemClient GetDataLakeFileSystemClient(string containerName, string dataLakeName, string dataLakeAccessKey)
{
    StorageSharedKeyCredential storageSharedKeyCredential = new StorageSharedKeyCredential(dataLakeName, dataLakeAccessKey);

    DataLakeClientOptions options = new DataLakeClientOptions(DataLakeClientOptions.ServiceVersion.V2019_07_07);
    DataLakeServiceClient dataLakeServiceClient = new DataLakeServiceClient(
        new Uri(string.Concat("https://", dataLakeName, ".dfs.core.windows.net")),
        storageSharedKeyCredential,
        options);

    DataLakeFileSystemClient dataLakeFileSystemClient = dataLakeServiceClient.GetFileSystemClient(containerName);
    return dataLakeFileSystemClient;
}

This code doesn't work for me. If I delete Priavet Link Private Endpoint from SourceDataLake then it works. Somehow Private Link and Private Endpoint doesn't work with this line of code:

Response<FileDownloadInfo> downloadResponse = await sourcefileClient.ReadAsync();

Do you have any Idea how can I solve this problem? or any better way to copy a blob from DataLake gen2?

uvr
  • 515
  • 4
  • 12
Kaja
  • 2,962
  • 18
  • 63
  • 99
  • Why not use AzCopy then? unless you have to read/modify metadata from each blob you don't need c# application – deathrace Jul 24 '21 at 09:34

1 Answers1

1

Sorry I cannot help with the current issue , but as you asked about other options , I think you can explore Azure data factory . https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage

HimanshuSinha
  • 1,650
  • 2
  • 6
  • 10