2

We're new to the Azure messaging service and Functions, we learned the basic concept of different message service like Storage Queue, Service Bus Queue, Service Bus Topic and Event Hubs we also read the fan-out pattern by microsoft but that works more like fan-out then fan-back, which means during the fan-out process, current function still hold up there until all resource fan-back.

A simple scenario would be, we are saving an object in azure function starter and after saved successfully into storage, we would like to send a common message to 3 different destination/consumers implemented/triggered with azure functions,

function 1 would be using for notification(signalR or email)

function 2 would be using for logging(Sql Server)

function 3 would be using for continuing business logic.

All of above azure functions will be working independently/isolated, and our azure function starter doesn't have to wait for any of these 3 functions, it should exit or terminate once the message sent out. Basically the concept would be one publisher with multiple consumer and can guarantee that message would be delivered to all 3 consumers.

Which of the messaging service would be recommend in Azure? I think we should pick one of those three messaging servce Storage Queue, Service Bus Topic and Event Hubs but there is limitation of Azure resource that we can refer...

LoekD
  • 11,402
  • 17
  • 27
Drex
  • 3,346
  • 9
  • 33
  • 58

1 Answers1

4

Simple - Azure Service Bus Topic.

We implement the same pattern at work. Once a message comes in we then send it out to the Topic which then gets picked up by different TopicListeners/Subscribers. Each subscriber then calls a third party service.

Here is documentation of the difference between Service Bus and Storage Queue. https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted

Paul Lorica
  • 703
  • 4
  • 6
  • Hi thank you for sharing!are you suggesting Service Bus Topic or Service Bus Queue? I am kind of confused as you mentioned topic first but the documentation was focus in Service Bus Queue. In addition, are your topic listeners/subscribers implemented using Azure Function? How would I ensure my message was delivered to all my subscribers and how would you handle the failed scenario with service bus topic? Can I store my messages somewhere so later on I can perform a retry if any of my subscribers having an exception? – Drex Aug 17 '19 at 02:17
  • @KevDing I am suggesting Service Bus Topic. Think of Service Bus Topic as a one-to-many publishers to subscribers, while Service Bus Queue is one-to-one. – Paul Lorica Aug 19 '19 at 14:20
  • Yes, everything was implemented in Azure functions using the Service Bus trigger. https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus We kind of just trust that the messages that the messages gets delivered to the subcribers. As for the failure scenarios, we actually set up our subscribers azure functions to send a message to a different queue which does the actual processing. That way if a failure occurs we can retry by putting it back onto the queue – Paul Lorica Aug 19 '19 at 14:31
  • thank you for your suggestion! What I was thinking is that, for example, my subscribe 1 and 2 completes its job successfully, while my subscribe 3 failed due to some exception, then I only need to retry with subscribe 3 not 1&2 again. So in your design, I assume you're creating 3 different queue(q1, q2, q3) just for retrying with subscribe 1, 2&3 if there is any failed case happen, correct? – Drex Aug 19 '19 at 14:37
  • 1
    @KevDing that is correct. But just to be clear subscribe 1,2,3 "should" always work since they are just a passthrough to queues 1,2,3. Azure functions that process queues 1,2,3 can fail and if they do, then they just either dead-letter or put the message back into the queue. – Paul Lorica Aug 19 '19 at 14:56
  • thank you so much for your advice! For using the service bus, I assume there is a pre-condition of publishing same message to different subscribers, but what if I need to publish different messages to different subscribers, would service bus topic still work here? I've opened a new ticket for explaining [my question](https://stackoverflow.com/questions/57560235/which-azure-messaging-service-to-choose-to-publish-to-multiply-subscribers-with), if you've ever worked in that scenario as well, could you please help me take another look at that? Thanks a ton! – Drex Aug 19 '19 at 16:01
  • Sorry for bothering you again, with your great help I designed my subscriber function that will be listening to two different message source, one is coming from service bus which should be the **normal business process**, the other is coming from queue storage which as considered to be **exception handler**, but I'm getting confused...If my function generates an exception, then it throws the exception and insert into **exception queue**, next time when the function re-processes the same message, it would then fail again and put into queue, which seems a indefinite loop event... – Drex Aug 19 '19 at 20:28
  • I think I might misunderstand your words, by putting the message back into the queue and let function re-process it, would it cause same error again and again? For example, my email server authentication failed, then with this design, my email function would be looping forever in authentication failed, put message in queue for retry, re-process but authentication fail, then put message in queue for retry...etc – Drex Aug 19 '19 at 20:31
  • 1
    yes, there is definitely potential for an indefinite loop. So there are two cases here. 1. Bad message - Will never be processed properly. What you need to do with these cases is to Dead-Letter the queue after X amount of retries. You can put in the message how many times it has tried to process. 2. Dependencies down - These you need to put back into the queue. Do a Scheduled message, and resend it say, like 5 mins from now. – Paul Lorica Aug 19 '19 at 21:33
  • that makes a ton of sense, thank you very much for sharing your knowledge! – Drex Aug 21 '19 at 00:30