Azure Data Factory Missing Blob Triggers

Question

I have created an ADF pipeline that should trigger when a blob is added to a storage container (say container1) and copy the blob to another storage container (say container2). All my blob names are alphanumeric with '-' (basically a GUID). I see that the ADF is triggered only a few times compared to the number of blobs in container1 (i.e if I have n files in container1, the ADF is triggered only x times where x<n).

I also observed that whenever the blobs created per second in container1 is high there are more missed triggers. I am not using any event batching in the event grid. My storage account is v2 BlockBlobStorage.

Is there a way I can resolve this?

Have you filtered certain file types, so the number of triggers is less? — Joseph Xu, Feb 04 '21 at 07:35
My blob trigger is only for blob created not blob deleted. But no blobs are deleted though. All the triggers are for created only — Sankar Mantripragada, Feb 04 '21 at 08:36
Hi @Sankar Mantripragada,I've created several tests but unfortunately I couldn't reproduce your issues. Can you tell us any progress now? — Joseph Xu, Feb 11 '21 at 08:27
Sometimes my storage can have around 100,000 blobs per hour, the triggers are missing if the inflow is that high. — Sankar Mantripragada, Feb 11 '21 at 11:27
Update: I changed the design not to use trigger pattern, instead kept a timer trigger to process blobs every hour — Sankar Mantripragada, Apr 05 '22 at 08:14

score 0 · Answer 1 · answered Feb 12 '21 at 06:38

0

I think it is difficult for us to get the correct answer in the community. We'd better move this issue to here. After MS's stress test to find out the possibility of bugs.

answered Feb 12 '21 at 06:38

Joseph Xu

5,607
2
5
15

score 0 · Accepted Answer · answered Mar 13 '23 at 11:48

0

Here they mentioned "There's no guarantee that all events are captured."

answered Mar 13 '23 at 11:48

Sankar Mantripragada

134
11

Azure Data Factory Missing Blob Triggers

2 Answers2