1

I currently have a 3rd party application pushing messages to a Lambda function through API gateway. The Lambda function needs to serialize, log, and push the message to another ESB that I have very little control over.

I'm trying to ensure that there is some kind of recovery mechanism in the case that the Lambda function is either at max load or cannot communicate with the ESB. I've read about Kinesis being a good option for exactly this, but the ESB does not support batching for my use case.

This would cause me to run into the scenario where some messages might make it to ESB, while others don't, which would ultimately cause the batch to fail. Then, when the batch is retried, the messages would be duplicated in the ESB.

Is there a way I could utilize the functionality that Kinesis offers without the batching? Is there another AWS offering that better fits my use case? Ideally I would have one message being handled by the Lambda function that stays in the queue until it is successfully pushed into the ESB.

Any tips would be much appreciated.

Thanks,
Matt

Matt Skinner
  • 11
  • 1
  • I feel that a SQS queue would be the best. See [this post](https://stackoverflow.com/questions/41097441/how-to-integrate-api-gateway-with-sqs) for an example of the first part of the solution. But Lambda and SQS can be a poor combination as you would then need to poll the queue to get data. Another possibility would be a DynamoDB table but it's a similar problem to SQS. Are you stuck with Lambda? What are your latency requirements? – stdunbar Jul 24 '17 at 22:03
  • what is the frequency of the push messages? is it real-time streaming or hourly, etc..? when you say max load you mean "AWS Lambda Resource Limit per Invocation" is reached or you mean API Gateway throttles? – Nicholas Jul 24 '17 at 22:41
  • @Nicholas The messages aren't being pushed very frequently. It's real-time, but I'd say there are usually no more than 10 messages being pushed in a minute. I'm just concerned that either the ESB will be unavailable, or in very rare cases the Lambda will be unavailable because of maximum invocations, so I need some kind of fallback. Some kind of queue where the messages are delivered only once would be ideal. – Matt Skinner Jul 24 '17 at 23:56
  • 1
    Do you need to preserve message order? – Joshua Jones Jul 24 '17 at 23:59
  • jjones has a good point - do you want to be time-travelable. Can these messages processed out of order – Nicholas Jul 25 '17 at 08:36

1 Answers1

0

Following might be of help to you:

1) setup api-gateway to log to sqs and 2) then set up a lambda function on that sqs queue to serialize, log, and push the message to the external endpoint.

For the first part: How to integrate API Gateway with SQS this will be of help. (as already mentioned in comments)

This article might help you more for second part: https://dzone.com/articles/integrate-sqs-and-lambda-serverless-architecture-f

Note that you can also choose what kind of trigger you would like (based on usecase)- cron based poll/ or event based, you also have control over when you are deleting from sqs in your lambda function. (you can also find the very basic code in lambda blueprint with name "sqs-poller").

Thanks!

Piyush Ahuja
  • 131
  • 3