-1

From last question about Azure messaging service by publishing one common message to multiply subscribers by using service bus topic. But we've encounter an issue that our message size is over-sized of the service buss limitation(256KB), so we changed our design by splitting the GIANT message into small piece of sub-message and send them to different subscribers, here is what I'd like to achieve:

Let's say we are saving an order object in azure function called SaveOrder and after order was saved successfully, we would like to send different messages(content) to 3 subscribers/consumers which is also implemented/triggered with azure functions,

subscriber/azure function 1 would be using for reporting - with message 1, e.g 

{
 'reportHeader':'Order Report for ID 012913',
 'report_notes':'A few additional report notes...'
}

subscriber/azure function 2 would be using for logging(Sql Server) - with message 2, e.g 

{
  'logAction':'Order Saved Successfully', 
  'logTime':'08/12/2019 12:38:12 AM',
  'logDescription': 'Order Id - 0139281',
  'log_notes':'A few additional log notes...'
}

subscriber/azure function 3 would be using for continuing business logic. - with message 3, e.g 

{
  'action':'Prepare product accumulation',
  'ProductList': {
    '813891':3
    '581231':1
  },
  'accumulation_notes':'A few additional accumulation notes...'
}

All of above azure functions subscriber will be working independently/isolated, and our azure function SaveOrder doesn't have to wait for any of these 3 functions, it should exit or terminate once the message publishes.

In this case, which of the messaging service should I choose in Azure for handling that? Would service bus still work here? As my understanding is that Service bus topic would send/publish "same/one" message content to multiply subscribers.

LoekD
  • 11,402
  • 17
  • 27
Drex
  • 3,346
  • 9
  • 33
  • 58

2 Answers2

0

You can have a separate topic or queue for each of multiple outbound streams. However you should probably publish a single message containing all necessary information to a topic and have three separate consumers do different things with the message.

For instance your function shouldn't be coded to generate an "email notification message" that's not its concern. It should publish an OrderSaved message, and if a service wants to subscribe to that and send emails, it can.

David Browne - Microsoft
  • 80,331
  • 6
  • 39
  • 67
  • thank you for your advice. Just to be clear, instead of sending 3 different messages, are you recommending on the publish side we should generate a "big" message including all the "detail" sub-messages then use service bus topic to publish, on the subscribers side for each subscriber we need to _filter_ on the incoming big message and take part in which needed? If so, we've thought about that, but we've encounter a challenge: there is a limitation for function app/service bus message size, and our message will exceed over the limitation if we include everything in one "big" message body. – Drex Aug 21 '19 at 12:56
  • Maximum message size: 256 KB for standard plan with using service bus – Drex Aug 21 '19 at 13:05
  • That'll be our initial idea of building a common message by using service bus topic, but because of the message size limitation in service bus, we've changed our design of split into different sub-message... – Drex Aug 21 '19 at 18:40
  • Why would the `SaveOrder` event have even 4K of data? That seems strange. – David Browne - Microsoft Aug 21 '19 at 18:45
  • Because we attached some more information to the output message, E,g, we're retrieving the production detail information in `SaveOrder`, and attached a few additional property in the message for each product, which increased the message size if production list is too large(we had case where # of products is over 50k), also for the notification and logging service, we attached additional info as well. – Drex Aug 21 '19 at 19:00
  • Another good practice here is not to include a whole lot of "master data" in your messages. Any downstream system that needs it should be able to look it up. – David Browne - Microsoft Aug 21 '19 at 19:58
  • Do you mean by storing the data somewhere in a temp location then having the next function look from there? If yes, we did think about that but that sounds like an extra step (storing/reading) which would lower the performance, we also think about using **EventHub** to send messages in batch to different subscribers, but not sure if that'd be necessary to do as **EventHub is designed to handle millions of data per second level** so it'd be over-required in our case, while service bus is less-required...If there is any other messaging service to recommend, please feel free to suggest any! – Drex Aug 22 '19 at 14:52
0

If message1 always only goes to function1, message2 only to function2, and so on, creating 3 Service Bus Queues would suffice. This simply transports the message from a sender to a receiver (function) (FIFO-style).

From SaveOrder, you'd send message1 to queue1, message2 to queue2 etc.

  • function1 receives messages from queue1.
  • function2 receives messages from queue2.
  • function3 receives messages from queue3.

Using topics and subscriptions will also work, by creating 3 topics with 1 subscription each:

From SaveOrder, you'd send message1 to topic1, message2 to topic2 etc.

  • function1 receives messages from subscription1.
  • function2 receives messages from subscription2.
  • function3 receives messages from subscription3.

The second scenario is more flexible, as it allows multiple receivers of the same (copy) message, by adding more subscriptions to the same topic.

Also, you can gzip the payloads of your messages to make transport more efficient.

LoekD
  • 11,402
  • 17
  • 27
  • thank you for your advice! Yes in this case msg_x goes to func_x only(1 to 1), On the 2nd option when you mean, _by creating 3 topics with 1 subscription each_, do I need to perform the same message to my 3 functions and based on the topic they will filter? I don't think so as also mentioned _you'd send message1 to topic1, message2 to topic2 _, but in this case, what'd be the difference of using **topic** with **queue**? My understanding is **topic** is using for 1 to many publisher/subscriber while **queue** is using for 1 to 1 publisher/subscriber, sorry I didn't get much there... – Drex Aug 22 '19 at 14:46
  • No, I mean that using a topic with 1 subscription (no filters) offers the same functionality as using a queue. – LoekD Aug 23 '19 at 05:34