2

I have created a storage event trigger to trigger a pipeline in Azure Data Factory. This trigger works when I manually put a file in the datalake. But when the file is uploaded from Azure Function, trigger doesn't work.

Below is my function to upload file in datalake.

public void UploadFileToDatalake(FileUpload file, string containerSasUri, string stage, string dir, ILogger log)
        {
            log.LogInformation("inside uploadFileTodatalake");
            UriBuilder sasUri = new UriBuilder(containerSasUri);
            DataLakeServiceClient dataLakeServiceClient = new DataLakeServiceClient(sasUri.Uri);

            string container = Globals._transientContainer;
            var fileSystemClient = dataLakeServiceClient.GetFileSystemClient(container);

            string pathFileDL = file.Name;
            
            string directory = dir;

            MemoryStream ms = new MemoryStream();
            StreamWriter sw = new StreamWriter(ms);
            sw.Write(File.ReadAllText(file.FullPath));
            sw.Flush();
            ms.Seek(0, SeekOrigin.Begin);

            file.Content = ms;

            DataLakeDirectoryClient directoryClient = fileSystemClient.GetDirectoryClient(directory);
            try
            {
                if (directoryClient.Exists())
                {
                    log.LogInformation(directoryClient.Name);
                    DataLakeFileClient fileClient = directoryClient.CreateFile(pathFileDL); // 0kb blob
                    file.Content.Position = 0;
                    fileClient.Upload(file.Content, true);
                    log.LogInformation($"{stage}: {file.Name} uploaded to {pathFileDL}");
                    file.Content.Close();
                }
            }
            finally
            {
                //fileSystemClient.Delete();
            }
        }

FileUpload is a model class I am using.

public class FileUpload
    {
        public string Name { get; set; }
        public Stream Content { get; set; }
        public string FullPath { get; set; }
    }

Thanks in advance.

Nikita M.
  • 21
  • 3
  • I realized my mistake here. FlieClient first creates an empty blob then writes into the blob. Which was causing a problem with trigger. So instead of FileClient, I used BlobClient and it got resolved. – Nikita M. Oct 01 '21 at 05:11

3 Answers3

1

I am tried with the blog for upload a file in azure blob using azure function.

Before creation of azure function, I was created the pipeline Storage Event Trigger to get trigger when the new file arrives/created on storage blob.

Tried with manually to uploading a file using azure portal. The event grid triggered.

After created and published the azure function on azure. Now run the azure function. It generates the random files on storage container and the expected Storage Event Trigger also run. I can see the Triggered output on the portal.

enter image description here

Delliganesh Sevanesan
  • 4,146
  • 1
  • 5
  • 15
  • Thanks for answering. I used BlobClient instead of FileClient. Changed nuget. And it worked. – Nikita M. Oct 01 '21 at 05:13
  • Thanks @NikitaM. I am also suggest to use BlobClient for further information [check this link](https://www.c-sharpcorner.com/article/create-container-and-upload-blob-using-azure-function-in-net-core/). – Delliganesh Sevanesan Oct 01 '21 at 05:25
0

By using ADLS FileClient, you should set Close option to true. Then the Storage Event will be sent yet another time after the upload stream was closed.

        var options = new DataLakeFileUploadOptions();
        options.Close = true;
Alexander
  • 1
-1

I realized my mistake here. FlieClient first creates an empty blob then writes into the blob. Which was causing a problem with trigger. So instead of FileClient, I used BlobClient and it got resolved.

Nikita M.
  • 21
  • 3