1

Our application consist of 7 microservices that have some intercommunication. Currently we're using simple storage queues that a microservice publish events to (the number of events is relative low). Then we have a azurefunction for each queue that might call another microservices. This is working fine for us right now the services uses about 20 queues with a corresponding function.

Now we need to handle an blobstorage event, and I did some googling and a started to get really confused. Suddenly there was a lot of questions:

  • Should we switch to Azure Event Grid
    • It handles blobstorage without any limitations (functions blobstorage trigger has some)
    • It allows for multiple subscribers (storage queues does not)
    • It has a lot of fuz - maybe this is the new recommended way
    • I like the idea of one central thing, but it reminds me a bit about biztalk...
  • Should I switch to Azure Service Bus
    • It has a nice tool (ServiceBusExplorer) for monitoring the queues and listners, and I could to a repost of any failed events
    • It visulizes my azure functions subscribers nicely
  • Should I continue with only storage queues
    • A bit difficult to monitor, but it works nice

I'll be really thankful for any advice or insights to this question.

Thanks

Janusz Nowak
  • 2,595
  • 1
  • 17
  • 36
Larsi
  • 4,654
  • 7
  • 46
  • 75

3 Answers3

4

EventGrid is great when you have notifications floating to multiple subscribers. Is that the case for you?

An example would be deferring messages. With queues you can defer a message, not with EventGrid. Whenever to choose Storage Queues or Service Bus depends on the specific requirement that you have. Do you need de-duplication? Or ordered delivery? If you do, Service Bus is the way. Otherwise Storage Queues is enough.

Sean Feldman
  • 23,443
  • 7
  • 55
  • 80
  • Thanks. Currently no need to defer messages. Currently only one subscriber. Currently no orderd delivery. But then, on the other hand - yes multiple subscribers would be nice for a few events, and duplication could be a problem.... Based on that - would your advice be to use all three? – Larsi Mar 08 '18 at 09:58
  • 1
    If pub/sub is important, Storage Queues is not a good option, Service Bus and Event Grid are better. Keep in mind that Event Grid won't give you de-duplication though. And you have to have a webhooks per subscriber for messages to be pushed. The charming part is that subscribers don't have to poll. Not even long poll. You'll have to look at your specific requirements and decide between the two what will be more suitable or if you want to introduce the two for what they are good at. – Sean Feldman Mar 08 '18 at 15:54
  • 1
    In addition to Sean's answer, The Event Grid is suitable to use for near-real-time Distributed Event-Driven Pub/Sub model across the regions, clouds and on-premisses systems. It's a push model with distributing of 10 millions events per second and region to the max 1000 subscribers per region. The events are delivered to the event sinks (subscribers) in the durable mode with a retry mechanism. It's perfect for event-driven serverless application architecture. – Roman Kiss Mar 08 '18 at 17:19
  • @SeanFeldman you can have eventgrid pushing events to a function or logicapp and then forward message to a servicebus queue also. I think with blob event, eventgrid is the way to go. the azure team will move to eventgrid implementation for blobtrigger (webjob/function sdk) – Thomas Mar 09 '18 at 07:42
  • @thomas exactly, but only if those are HTTP triggered Eventually any resource on Azure will be emitting events, but that's not helping if you can not poke a hole for inbound calls. BTW, integration between EventGrid and Service Bus is already here, but for premium tier only. Eventually will be in standard bas well. – Sean Feldman Mar 09 '18 at 14:40
3

First of All, I would like to recommend these two articles, it will clarify most of your doubts about these services:

Regarding Event Grid, it acts like a bridge between the publisher and the subscriber, where publisher will send messages and forget whether it has been processed or not, and the Event Grid will handle the retry if the receiver\subscriber does not acknowledge that it was processed successfully.

As you mentioned, storage queues has limitations, as such blob triggered functions, and maybe Service Bus, but it will depend on your design requirements. I would like to point out some things you might consider before moving to Event Grid.

  • Storage queues & Service Bus does not care about your message schema, in Event Grid you have to create a custom event based on their schema to wrap your event, so the publisher and subscriber has to understand Event Grid for that, not that is a big deal, but now you have both sides coupled to Event Grid.

  • If you want to send the event straight to your micro-service, you have to implement the subscription validation in your service, otherwise the service won't be able to receive the events

  • Event Grid only retry the delivery of your messages for 24 hours, if your service is down or not process the message correctly for longer than 24h, it will make the event dead. Currently, there is no way to query dead messages. Storage Queues and Service Bus are configurable how long you keep the message and it can be kept for many days.

  • Your service web-hook must acknowledge the receipt(http 200 or 202) of an event within 60 seconds, otherwise it will consider failed. If your operation is longer that that, you should send it to a queue and handle the locking from your service.

Probably there are more limitations, but these are the ones I remember right now that might change anytime soon, I think Event Grid is a great technology still on early days, and there is much to improve, I would recommencement only as a hub for Azure management events, I don't think it is ready for use as an application integrator.

Regarding your comment for queue manager, for Service Bus your have the Service Bus Explorer, and for Azure Storage you have the Azure Storage Explorer, where you can check the messages in the queue, is not the same as Service Bus, but helps.

Diego Mendes
  • 10,631
  • 2
  • 32
  • 36
  • 2021 update : You do not have to use the EventGridEvent schema for EventGrid events, not that it is a bad thing to use, but you can also use Cloud Events v1.0 as per https://learn.microsoft.com/en-us/azure/event-grid/cloud-event-schema Seperate point, if events cannot be delivered in 24 hours to a subscriber, they can be deadlettered to a Storage Account (Blob container) and you can log the Diagnostics to a workspace, or trigger further events or alerts from that action. – Peter Jun 17 '21 at 09:00
0

It very much depends on how are you consuming the queue messages, you can take a look at this comparison: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted

If you don't need ordering and if you don't have a strong limit on message volume, size or TTL, you can stick to storage queues.

Horia Toma
  • 1,099
  • 2
  • 17
  • 29