Retry logic for integrating with an external API

Question

Imagine we have an asynchronous, event-driven microservice architecture in place. As part of some business function, one of our services needs to reach out to a third party API. This API happens to be in the form of a synchronous REST endpoint.

How would we handle failures in cases when the third part service is down, for example? I have an approach which might work, but need some feedback please:

The service responsible for making calls to the third party would publish these requests as events to a dedicated Kafka topic. It will then consume it's own messages from that topic - in successful calls it will simply commit the offset of that message and all is OK. In unsuccessful cases, I assume we need to wait 'n' seconds/minutes before trying again. If the request is still unsuccessful, we can publish these errors to some dead letter queue and handle them later.

A few questions came to mind about this approach:

• Is it a normal practice to create a Kafka topic simply for storing required requests to third party API's which might fail? Just feels like overkill to me, even though I can't think of a better alternative at the moment.
• Is a service consuming it's own messages a common practice in asynchronous architectures?
• Would we not also block the Kafka queue while the offset of the message is not committed when an API call fails? How can we work around this?
• Just out of curiosity, how would this be handled in a more traditional system where asynchronous communication doesn't exist? Just store the failures in a database and decide how to handle them ad-hoc?

Thanks!

Answer 1

You would only use Kafka here if you care about ordering of requests. You'd consume an event, and make each request, but have retry logic in case of failure (not consuming next offsets in the same partition). On a successful case, commit that offset, as you said.

If the external API is just fire-and-forget, and you don't care about tracking failures you can skip the dead letter topic.

service consuming it's own messages a common practice

Nothing prevents it. It's very common with stateful processing, such as Kafka Streams.

how would this be handled in a more traditional system

Many systems have request-reply HTTP clients. Any web app, for example, would just have the user retry the request on their own. The only reason you'd persist the requests into a database or Kafka topic is if you were to retry the event on your own backend.