0

Context:

  1. In Azure function with EventHubTrigger, I save data mapped from handled event to database (through the Entity framework). This action performs synchronously
  2. Trigger a new event about successful data insertion using event hub producer. This action is async
  3. Handle that triggered event at some other place

I guess it might happen that something fails during saving data, so I am wondering how to prevent inconsistency and secure that event is not sent if it should not. As far as I know Azure Event Hub has no outbox pattern implemented yet, so I guess I would need to mimic it somehow.

I am also thinking about alternative and a bit smelly solution to make this publish event method synchronous in step 2 (even if nature of the event-driven is to be async) and to add an addition check between step 1 and step 2 - to make sure that everything is saved in db. Only if that condition is fulfilled, event is going to be triggered (step 3).

Any advice?

nikolan
  • 3
  • 3
  • I'm not sure that I understand the distinction that you're making when calling out `sync` and `async` in this context. Are you referring to the nature of the implementation itself _(`Foo()` versus `await FooAsync()`)_ or is there another meaning to them in this context? – Jesse Squire Jun 11 '21 at 13:39
  • When you say `trigger a new event` in step 2, are you implying "publish an event to Event Hubs using a producer client" or "I want something to observe the record creation in my database and signal another function instance when that happens?" – Jesse Squire Jun 11 '21 at 13:40
  • @JesseSquire yes, I am referring to the nature of the implementation. At the time being, InsertToDb method is implemented to be synchronous and SendEvent is async. So, my assumption is that if I have two methods in sequence and first one is sync and second one is async, the last one will be executed asynchronously. So, no matter what happens with this db insertion, this event would be sent. `trigger a new event` refers to "publish an event to Event Hubs using a producer client", so nothing like db trigger or such. So, if event cannot be sent, I would end up with inconsistency. – nikolan Jun 11 '21 at 17:08
  • That's not quite how async works; there's no concept of "defer until some unknown point in the future." The async call happens deterministically at the time your code makes it, but may pause/resume/move threads. When you `await` the async method, you have a deterministic point that your code will resume when that call completes and you'll know if an exception occurred in that call. In your scenario, If you call your sync method and an exception occurs, you can chose not to invoke the async method. – Jesse Squire Jun 11 '21 at 19:40

1 Answers1

0

There's nothing in the SDK that would manage distributed transactions on your behalf. The simplest approach would likely be having a column in your database that allows you to mark when the event was published, and then have your function flow:

  1. Write to the database with the "event published" flag unset; on failure abort.
  2. Publish the event; on failure abort. (the data stays in written)
  3. Write to the database to set the "event published" flag.

You'd need a second Function running on a timer that could scan your database for rows older than XX minutes ago that still need an event, which then do steps 2 and 3 from your initial flow. In failure scenarios, you will have some potential latency between the data being written and the event published or may see duplicate events. (Event Hubs has an at least once guarantee, so you'll need to be able to handle duplicates regardless.)

Jesse Squire
  • 6,107
  • 1
  • 27
  • 30
  • Thanks @Jesse Squire. Does it make sense to follow the same approach for all events, not only ones that have some CRUD operation before publishing them? And do you have any insight how often an event is not published in Azure world? Or is it worth it to implement this additional check (to have new dedicated table for managing events' statuses and job/scheduler)? – nikolan Jun 12 '21 at 22:05
  • If you `await` the call to `SendAsync` and do not see an exception, then the broker has accepted the send and assumed responsibility for the messages. That's your guarantee that the events have been published. It will take a non-deterministic period of time for them to appear in partitions so that they can be read, but they're safely published. – Jesse Squire Jun 14 '21 at 13:43
  • If the `SendAsync` call fails, that means all retries have been unsuccessful and either there's an extended issue with the service/network or a data issue _(such as the event being too large to ever publish.)_ It's the extended outage case that you're guarding against with this pattern. It gives you a basic long-term retry for publishing in the face of an extended transient problem in a less complicated way than a full eventually-consistent distributed transaction, such as that offered by the [saga pattern](https://docs.microsoft.com/azure/architecture/reference-architectures/saga/saga). – Jesse Squire Jun 14 '21 at 13:47
  • As far as frequency, the Event Hubs service has an excellent track record of availability and reliable operation. Beyond that, it's difficult to speculate due to how variable network reliability can be. If you're running in Azure Functions and publishing to an Event Hubs namespace in the same region without a ton of VPN infrastructure in the way, it's safe to say that you're unlikely to see many failures. If you're crossing-regions or publishing across other network boundaries, you'd have to look at the stability of your network (and DNS, because it's always DNS!) to understand. – Jesse Squire Jun 14 '21 at 13:51
  • Thanks a lot for clarification @Jesse Squire !!! Then, I assume this pattern could be applied to each and every event in app...including these steps: 1) create an instance of EventToBePublished class/entity with properties: EventHubName (time triggered azure function should know where to send unsent event), EventJson (serialized event object) and EventPublished (false by default) and save it to db 2) check if it is saved to db (using event unique id) 3) if it is saved, then publish the event 4) set EventPublished to true and update related row in db – nikolan Jun 16 '21 at 08:04
  • Finally, time triggered azure function would frequently check if there is any candidate in EventToBePublished table for publishing. If so, function picks up the entire event (saved previously as json in db) and knows exactly where to send it thanks to event hub name (assume that all hubs are under the same event hub namespace). – nikolan Jun 16 '21 at 08:09
  • Sounds like that would work. Depending on what your event consists of, I wonder of the necessity of storing something heavy like JSON, though that would be contextual to the application. I had assumed it was a lightweight shoulder-tap style event for a claim check pattern implementation. If that's not the case, you may want to consider doing it all as a single workflow - db update, event publish, db update to record the event publish. That would allow you to remove latency in the success case and let your timer-based function be your fallback for exception cases. – Jesse Squire Jun 16 '21 at 13:29
  • Yes, I agree that saving JSON in db might be too much (but what if I choose Cosmos db? Would it be fast enough to deal with JSON?). Btw, what do you think if I use CustomEvents container (extend CustomEvents with event body and isPublished properties) in Application Insight for this purpose? This way I would be able to set easily two things: 1) delete records after some time 2) craete time trigger to pick up unsent event. What do you think about that since this service is more for logs and telemetry and not suppose to be something that performs action such as resending events? – nikolan Jun 17 '21 at 07:46