3

I have a dynamoDB stream and lambda trigger for a table. The lambda trigger basically sync dynamoDB table to DocumentDB.

What if, DocumentDB is down for more than 24 hours. How can I put back the all the activity(put, delete, update) happened in dynamoDB back to the stream so that lambda trigger can access the records and sync the data to DocumentDB.

I see that dynamoDB stream keeps the record for maximum of 24 hours.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Avadh Gupta
  • 49
  • 1
  • 1
  • 4
  • You can't access records older than 24 hours. The only way is to backup them. So your lambda would have to not only upload it to DocumentDB but also store them for backup purposes. – Marcin Jan 20 '21 at 03:59

1 Answers1

5

By default its not possible unlike Regular Kinesis, which has max retention of 7 days, Kinesis behind DynamoDB has max retention of 24 hours and messages will be discarded after it exceeds max retry attempts and deleted after 24 hours.

So, we need to build an exception handling process, one such methods

  • Create an SQS Queue with higher MessageRetentionPeriod (max 14 days) and set a RedrivePolicy maxReceiveCount on no of times to retry.
  • Setup Destination on Failure on Lambda to SQS.
  • Same Lambda can be slightly modified to read either from Kinesis or from SQS or a different Lambda can be used to read from SQS.

Throw error back from Lambda when it fails to write to DocumentDb. This will send record back to Kinesis/SQS. This way we can get away up to 14 days. We can add a DLQ on SQS to another SQS too, which can send left over messages after 14 days to DLQ, with destination to a persistent storage.

Balu Vyamajala
  • 9,287
  • 1
  • 20
  • 42
  • Probably would need to address the issue of no order and duplicates in SQS, as this can be important to backup DynamoDB records. Probably have to use FIFO? – Marcin Jan 20 '21 at 05:05
  • 1
    Ideally we should build Lambda logic to be idempotent. ex: we can check last event timestamp which updated documentDb and ignore updates from older events, etc. But if that is not possible, yes, FIFO queues. – Balu Vyamajala Jan 20 '21 at 05:09