0

Trying to design a solution for error handling. We have a lambda that receives data from an sns topic and sends data to a legacy service that has been known to be unavailable at times.

When the legacy service is down I want to send the messages to a dyanmodb table to be replayed.

I want to use a circuit breaker pattern. So at the minute I am thinking of spinning up a service that will constantly poll the legacy service then some pseudo code that looks like this

If (legacy service changes from dead to alive){
     Send all entries from dynamo to sns topic;

//This will trigger the lambda again which will hit legacy service which we know is now up
}

The thing is, we like using serverless technologies and not sure I can have a serverless service that constantly polls, it makes sense for that to run on a server.

I am looking for a nice way to do this so I am wondering is it possible to configure dynamodb to poll the legacy service and on the condition it changes from dead to alive populate the sns topic. Or any other solutions using serverless technologies.

P.s I don't like the idea of running a lambda in intervals to check the dB as we could miss some down time, also reading data from dB and sending to sns could be a lengthy operation.

Update: been reading into circuit pattern more and realise I don't need to constantly poll I can just check amount of failed calls in last XX seconds in my dynamodb table so a new question has arose, can I send message from dynamodb to sns depending on a condition on one of its entries. Eg. FailsInLastMinute changes from 3 to below 3 we send all the messages from a column in dynamo to sns or do I need a service for this part

AnonymousAlias
  • 1,149
  • 2
  • 27
  • 68
  • When the legacy service wakes up, could they send a trigger? – Nghia Do Sep 06 '20 at 02:33
  • Not really sure because it's not a planned shutdown and also want to cover for the case it's not down but still can't be reached for some reason. – AnonymousAlias Sep 06 '20 at 05:33
  • I mean whenever the service ups and works, could it perform any trigger such ad drop a file , send a msg, ... – Nghia Do Sep 06 '20 at 11:45
  • Yea it's possible but what if we can't reach the service because something happens with security groups or something. It won't come back up and trigger anything when security group issue is fixed. It was always running just couldn't be reached. – AnonymousAlias Sep 06 '20 at 14:42

1 Answers1

2

I don't think DynamoDB can do this, it's a database after all not an integration platform.

That said, a possible solution would be to use DynamoDB as a queue between SNS and the legacy app using DynamoDB streams. Any message from SNS gets inserted into DynamoDB using a Lambda. DynamoDB streams then triggers another Lambda that sends the message to the legacy app.

If the legacy app is down the Lambda function generates a failure as it cannot connect. DynamoDB will then retry the Lambda until it succeeds.

Note that you are probably better off using an SQS queue with fifo enabled. This will do the same but without the overhead of DynomoDB.

sihaya
  • 1,357
  • 9
  • 12
  • The problem with a fifo sqs queue is that it turns everything into a synchronous flow? Do you know with dynamo streams is it possible to do something like if the value of a string attribute="connectionOpen" then send all rows of data in the database that have an attribute status="failedMessage" to an sns topic? – AnonymousAlias Sep 06 '20 at 20:31
  • Like I don't want to constantly try to hit the legacy service until it is up with every message.i was thinking once I get confirmation that the legacy service is back up I will update the connectionStatus in dynamo and then that would trigger a stream of all the payloads that would of failed on the legacy service if that's possible? – AnonymousAlias Sep 06 '20 at 20:33
  • When the connection is restored you could update all the rows with the "failedMessage" attribute. This will trigger the streaming Lambda which should then deliver the messages to the legacy service. I would still consider a solution with SQS and use delay queues to prevent calling the legacy service too many times when it is down. You could also maybe programmatically stop the processing of the queue if calling the legacy service fails for x times. – sihaya Sep 06 '20 at 21:05
  • So for this I was thinking on using the circuit breaker pattern.if we have n amount of failed attempts in the last n minutes then we don't even try the legacy service we just send directly to dynamodb. this is repeated until the connection is open, once it is I want to send all the dynamo entries (bar the one that counts the failed messages) back into an sns topic that will trigger the lambda the sends to the now alive legacy service – AnonymousAlias Sep 06 '20 at 21:24
  • So are you saying I can look for a specific update like specifically updating the failedMessage attribute and this will trigger the stream to sns but stream won't be triggered for any other updates like normal error messages going in? – AnonymousAlias Sep 06 '20 at 21:26