0

We have the following setup:

Event Hub —-> Azure Function —> API.

We are trying to come up with a solution for Dead lettering with Event Hub. Event hub doesn’t come with Dead Lettering, hence we need to build our own solution.

We came up with 2 options :

  1. An Azure Function with Event Hub trigger, will have try catch block, it does have a retry policy to call the Api, if the call fails after all the attempts , we will store the event, metadata( offset of the event) in azure table storage so that we can reprocess this event from table storage using DLQ re-processor.

  2. A Durable Function with Event Hub trigger. Will have an orchestrator and activity functions. If the processing an event fails with an error, we can retry the activity functions since it’s persisted.

I would like to know which option is a better choice here. I am not able to understand how the Deterministic policy of Durable Functions come in to play here.

We tried option 1 and have a feeling that we are building a state machine in azure storage. We are expecting if azure durable functions built in features will help us in handling dead lettering and resiliency.

vamsi krishna
  • 31
  • 2

1 Answers1

0

Both of your options are, at the core, the same implementation, apart from the fact that Durable Functions has everything that you already need.

Here are some things that should help you setting everything up with Durable Functions

1. Leverage Automatic Retry with Activity Functions
This will ensure that the first run of each orchestration is run to completion even with intermittent issues.

2. Build Dead Lettering on top of Instance Management APIs
Since it is always possible something else is wrong and requires fixing (which is why you are asking this question ), Durable Functions already comes with APIs that allow you to query failed instances and re-run them on demand.

Depending on your scenario, you could invoke these APIs on a schedule, or an event.

3. Use the Netherite Storage Provider, if required
Based on the scale of your application, you could consider using this storage provider instead of the default Azure Storage Provider.

But before making the move, you might want to assess the scale and cost benefit of the switch. You can read more about the engine in their official docs.

PramodValavala
  • 6,026
  • 1
  • 11
  • 30