2

I want to create an EventGrid via Arm deployment to ingest to ADX cluster once a all blobs in the container were added . (it's a bunch of blobs created each certain time interval)

I want to make sure that the event is raised only when a blob is created and its content has been finished written to. (not just a "blob created file").

The Storage is Data Lake Storage V2 which mean it has "Hierarchical namespace"

I'm looking at this link:

https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?tabs=event-grid-event-schema

I'm confused between the: List of events for Blob REST APIs section and List of the events for Azure Data Lake Storage Gen 2 REST APIs section.

Which one is relevant to me and can I achieve my goal or is there a way which I can circumvent it to achieve my goal??

Ace
  • 831
  • 2
  • 8
  • 28

2 Answers2

2

I have seen various ways to circumvent this, some good, some bad. Just depends on what works for you.

  1. You could Wait for a specified amount of time when the BlobEvent trigger is first triggered. If you have small files that you know should never take more than 2 or 3 minutes then this is an option. If you don't know if it will take 5 minutes or 30 minutes for the Blob to be fully loaded, then this is a terrible option. I personally dislike this option, but it works for some. The Wait Method

  2. Stage the Blobs using stream or direct copy, then move them over via direct copy that does not fire the trigger before it's fully committed. I personally liked this method because it allowed Blobs to be moved into Staging, then Metadata was set on them, then Blob Copy command was executed on them to copy over with the metadata included. (FYI, ADF has functionality to set metadata within the copy activity, but this same principal can be applied to Azure Functions, Powershell, C#, whatever code is putting the blobs out there.)

Stage Blob, perform action, then copy to final container destination

  1. You could theoretically use an UNTIL activity in ADF to check the status of the incoming Blob and once it is either no longer growing, has a particular status, maybe a particular last modified (something like these) then you can continue on in your pipeline. This method I have not tested, but I expect that it would be possible to continually get the metadata of the file and use that to determine if the pipeline is ready to continue on.
Trent Tamura
  • 1,079
  • 6
  • 16
  • I think I would use a version of wait which just mean I would use a schedule trigger and instead of waiting 1 hour between runs, I would wait 1:10 since I know there's no more than 7 minutes delay. – Ace Jul 14 '21 at 19:15
  • You could use a schedule trigger if you know when the Blobs are expected to begin loading, but if you don't know when the blobs are supposed to begin loading or it's random times, then a Storage Event trigger is the way to go, and just have a WAIT activity be the first activity in your pipeline, in your case waiting for 7-10 min sounds like. Glad I could help! – Trent Tamura Jul 14 '21 at 19:19
  • Accept as answer if it meets your needs! Thanks – Trent Tamura Jul 14 '21 at 19:20
0

I have made a test to see if data is really not being ingested to the cluster from the type of storage and indeed no data is ingested. It's a copy activity from ADF. I assume because the event is trigger for just blob creation. I would probably use ADF.

Ace
  • 831
  • 2
  • 8
  • 28