Is azure compute slow in pushing events to Azure EventGrid? Is there a way to get events quicker?

Question

I have an azure function app with runtime v2 and language C# using EventGridTrigger. The function is subscribed to all events originating from an azure subscription. My main goal is to process events about a virtual machine's start and stop actions, and then perform certain actions when those events are received.

However, I am noticing delays of about 30 - 120 seconds between the EventTime noted in the event, and the time it is received in the function app. I verified that this is not a cold start issue by making sure I restarted the app just before triggering events. It sounds more like an azure compute issue to me.

For example, I restarted my app. Then I hit the Start button for a VM in the azure portal. After sometime, when the vm starts, azure compute sends an event to event grid, which my function app receives and just logs the event along with the event time. (see picture below).

Note that there is about 107 seconds delay between EventTime (12:44:05 AM), and when it was received and logged by the function (12:45:52 AM).

To dig deeper, I tried to check if I see similar delays when I push my own events. I created a custom topic, and subscribed another function to this topic. And then I pushed events to my custom topic using Azure Cloud Shell. In this case, I can see that my function received the event almost instantly. The delay is less than 1 second.

It indicates to me that EventGrid per se is quick, but the azure compute (VM infrastructure) is slow in pushing events. For example, if it creates an event object with EventTime t, but then publishes it at (t+t1) time, then of course EventGrid can do nothing about the t1 delay that got introduced before the event reached EventGrid.

Is my understanding (/speculation) here correct? Are there ways to get notified more quickly (<10 seconds delay)?

Likely the service emitting the event issue and not EventGrid. The best way would be to talk to the product group (Compute). Try to ping via Azure Support on Twitter. — Sean Feldman, Aug 10 '19 at 15:23

score 2 · Accepted Answer · answered Aug 10 '19 at 19:45

I agree with @SeanFeldman. I have tested for five VMs in different locations and AEG Subscriber for Service Bus queue.

I have used the REST API for start and deallocate a virtual machine. For status of the VM has been used the REST Get Instance View request.

The following is the result:

Example for start VM:

Instance View response statuses:

"statuses": [
    {
      "code": "ProvisioningState/succeeded",
      "level": "Info",
      "displayStatus": "Provisioning succeeded",
      "time": "2019-08-10T11:18:15.8833099-07:00"
    },
    {
      "code": "PowerState/running",
      "level": "Info",
      "displayStatus": "VM running"
    }
  ]

interesting (available) times:

"time":          "2019-08-10T11:18:15.8833099-07:00"
"eventTime":     "2019-08-10T18:18:29.5645489Z",
EnqueuedTimeUtc:    8/10/2019 6:19:54 PM

The above timing shows, that the AEG event message has been created after ~15 seconds (eventTime) when the VM running (time). The AEG pushed the event message to the subscriber queue after ~100 seconds (EnqueuedTimeUtc).

Example for deallocate VM:

Instance View response statuses:

 "statuses": [
  {
    "code": "ProvisioningState/succeeded",
    "level": "Info",
    "displayStatus": "Provisioning succeeded",
    "time": "2019-08-10T11:51:11.9832611-07:00"
  },
  {
    "code": "PowerState/deallocated",
    "level": "Info",
    "displayStatus": "VM deallocated"
  }
]

interesting (avalaible) times:

"time":         "2019-08-10T11:51:11.9832611-07:00"
"eventTime":    "2019-08-10T18:51:24.7467938Z",
EnqueuedTimeUtc:   8/10/2019 6:52:58 PM

Note, that the above delay can be saw also in the VM/Active log portal page.

Is azure compute slow in pushing events to Azure EventGrid? Is there a way to get events quicker?

1 Answers1