3

I'm working on a NestJS project that receives data from SAP MII and then send it to EventHub. Unfortunately, EventHub supports a maximum of 1MB (https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas), and in my case, SAP MII sometimes returns 4MB+ and I still need to send it to EventHub.

I have a few ideas on my mind, but I'm not sure if there's a better approach to it or even if there's a way to change EventHub size limit.

Sandra Rossi
  • 11,934
  • 5
  • 22
  • 48
Lucas
  • 491
  • 6
  • 21
  • 3
    The most common pattern is to store a large payload in a blob (e.g. your 4MB+ items from SAP) and then store a URI to the blob, inside your event hub message (along with any other info needed to process the payload). – David Makogon Jan 06 '22 at 14:30
  • @DavidMakogon But wouldn't that imply that whoever receives it from EventHub should make a request to where I stored that blob (like on my application db)? I'm not very azure-savvy, so I might be getting the wrong idea, sorry. – Lucas Jan 06 '22 at 14:45
  • 2
    yes - correct: read message from event hub containing blob uri, then perform whatever you need to do, by getting SAP data from that blob. Yes it's an extra call, but should be relatively fast, if storage is in the same region. Also - I didn't say 'application db' - if you store your 4MB data in a database, you will likely run into different scaling issues. I suggest you start with some documentation on Azure Blob storage to get a better idea of how it works, and how it can help you (and it supports objects far larger than 4MB...) – David Makogon Jan 06 '22 at 14:48
  • Keep in mind this pattern will require maintaining access policies for the app that writes to the storage and all the ones that would read the URI from Eventhubs to download the file – OneCricketeer Jan 06 '22 at 14:54
  • What SAP software is it? – Sandra Rossi Jan 06 '22 at 15:20
  • @DavidMakogon There's a problem in doing this, I have 4 services... The first one gets data from the SAP sends it to EventHub ... The other 3 only "know" EventHub and each other, so the NestJS service that communicates with SAP can't communicate with anyone else besides SAP and EventHub, so if I saved it to a DB, the other 3 services would not be able to get it EDIT: Just read the whole thing, I didn't pay attention to the Azure Blob part, I'm so sorry... I'll take a look into it, thanks – Lucas Jan 06 '22 at 17:09
  • 1
    @SandraRossi SAP MII – Lucas Jan 06 '22 at 17:09
  • 2
    The pattern that David is describing is known as "claim check"; some further examples and discussion can be found here: https://learn.microsoft.com/azure/architecture/patterns/claim-check – Jesse Squire Jan 06 '22 at 19:51

0 Answers0