1

There are a couple other topics out there, but none with solutions or none pertaining to Python Functions.

Background:

  • EventGrid-triggered, Python Azure Function
  • EventGrid messages created only when a blob is uploaded to a given Storage Account
  • Function receives message, downloads blob from message URL and does "stuff"
  • Function can run for several seconds/minutes (up to 120 seconds for large blobs)

Example of issue:

  • 4 files uploaded to blob container in correct Storage Account
  • Function successfully triggered 4 times, by 4 separate EventGrid messages
  • Function downloads blob from URL in each message, does "stuff"
  • ~55 seconds later, 4 more EventGrid messages trigger the Function all over again (for the same 4 files!)
  • Everything repeats

This happens multiple times resulting in 12 Function executions for 4 files:

  • And corresponding output from the "stuff" the Function does!

enter image description here

It gets ridiculous when 2500 files are uploaded to the Storage Account!

Seems like I need to adjust the EventGrid retry timing. But I don't see a setting for this in the Portal:

enter image description here

How do I prevent this behavior?

EDIT 1: Then today... no issue with 16 files uploaded... why is this Function inconsistently being triggered by EventGrid?

enter image description here

EDIT 2: And again today... for no reason, ~an hour later... EventGrid fired off a bunch more triggers though NO MORE FILES have been uploaded to the storage account.

enter image description here

Here are the EventGrid stats for 16 files being uploaded to storage account.

  • You can clearly see the numbers are all over the place with in some cases, ~1hour between retries.
  • Looks quite arbitrary to me

enter image description here

EDIT 3: For anyone interested...

ericOnline
  • 1,586
  • 1
  • 19
  • 54

1 Answers1

1

Based on the doc, the subscriber (such as your EventGridTrigger function) needs to send a response back to the AEG within the 30 seconds otherwise the message is queued for retry.

Note, that the event is removed from the retry queue when the AEG received within 3 minutes successful respond from the delivery destination endpoint (subscriber). If the deadlettering feature is on, the event is removed from the retry queue also when the respond failure code are 400 or 413.

Based on the above and your long running subscriber, the AEG sent a duplicate event within the 3 minutes.

I do recommend use a Push-Pull pattern in your solution such as delivery an event to the storage queue.

Roman Kiss
  • 7,925
  • 1
  • 8
  • 21
  • What do you mean by "Push-Pull" pattern? Could you point me to an example? – ericOnline Nov 04 '20 at 17:11
  • Notice in the logs above... the Function in this case is running from ~1sec to 4.5secs. This is WELL WITHIN the 30sec retry window. – ericOnline Nov 04 '20 at 17:15
  • Then today, each Function run is ~1 - 3.5secs. This time WITH NO RETRIES occurring? Is EventGrid a production-ready trigger for Azure Functions that run less than 30secs? – ericOnline Nov 04 '20 at 17:49
  • 1
    The delivery *Push-Pull* pattern in the loosely decupled Pub/Sub model allows to deliver (store) events to the resource entity (such as storage queue, event hub, service bus, etc.) in the real-time and than pulled up their based on the needs. Note, that the AEG requires to receive a response (ACK/NACK) from the delivery process based on the delivery and retry policy. One more note, in the *Push-Push* pattern (such as EventGridTrigger) the response time can be depended also when the function "cold started" – Roman Kiss Nov 05 '20 at 07:28
  • 1
    For test purposes: turn on a deadlettering feature with "maxDeliveryAttempts":1,"eventTimeToLiveInMinutes":30 for details about the *lastDeliveryOutcome* – Roman Kiss Nov 05 '20 at 14:13
  • RE: Cold start: I'm using Premium App Service Plan (P2V2). From what I understand, there is no cold start for this level of plan. – ericOnline Nov 05 '20 at 18:54
  • RE: Deadlettering, maxDelivAttempts, eventTTL: Will test this after support call with MS. I don't want to alter any variables at the moment. – ericOnline Nov 05 '20 at 18:55
  • @ericOnline, I’m facing the same issue. Did you resolve your problem? – Pine Code Jan 23 '21 at 23:17
  • Yes I did! I had to add an advanced filter to the EventGrid trigger: `data.api` `String contains` `FlushWithClose`. See the `Note` here https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage#list-of-the-events-for-azure-data-lake-storage-gen-2-rest-apis – ericOnline Jan 24 '21 at 22:43
  • @ericOnline how adding filter solved this issue ? Does it improve event grid trigger function ACK to AEG ? – Pintu Jun 12 '21 at 10:34
  • @Pintu, before the advanced filter was in place, EventGrid would trigger for each API call of the file upload. In my case, when Azure Data Factory uploads a file, it uses the `AppendFile` api for each chunk of the file. This triggered EventGrid and the Function each time. Instead, I filtered EG to only trigger when the `FlushWithClose` api is used. This is the last api called by ADF when has finished a file upload. – ericOnline Jun 22 '21 at 16:52