0

I have a DynamoDB table that has TTL and DynamoDB Streams enabled/configured.

I want to implement a lambda function that will read the TTL deleted data from DynamoDB Streams and forward it to maybe Kinesis Firehose or S3 (this I need to decide which one is better considering cost).

Is there any flag/property that will help to identify that the TTL deleted record is already being read/processed by some lambda function? For instance, I have 10 records in DynamoDB Stream, a lambda function has read 5 records and did some processing on it (forwarding it to Kinesis Firehose or S3). So, I want to identify those 5 records which were processed.

Reyan Chougle
  • 4,917
  • 2
  • 30
  • 57
  • 1
    not sure if I understand your requirement correctly. Might this help: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/time-to-live-ttl-streams.html – Korgen Jul 13 '21 at 08:02
  • @Korgen - that's exactly the right docs, if you expand it to an answer you'll get my upvote ;) – Maurice Jul 13 '21 at 08:08
  • What do you mean by "is already being read/processed by some lambda function"? – Marcin Jul 13 '21 at 08:16
  • @Marcin I have updated my question about what I mean by read/processed by some lambda function – Reyan Chougle Jul 13 '21 at 08:36

3 Answers3

2

you can use the userIdentity field in the stream record to identify which record came from DynamoDB's TTL deletion.

See the details here: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/time-to-live-ttl-streams.html

Korgen
  • 5,191
  • 1
  • 29
  • 43
  • This I know how to identify if it is TTL deleted. But I want to know if the record is being processed by any lambda function. I have updated the question with example user case – Reyan Chougle Jul 13 '21 at 08:38
  • Hi @Korgen , Sorry to hijack this thread - but do you know if the userIdentity field is only populated for TTL deletion? I haven't tried TTL deletion just yet, but the userIdentity is always null when I insert or modify data in dynamodb (via the application or via dynamodb console). I posted a question about this, but no response just yet. Hope you have better ideas than me. – dmc Mar 16 '22 at 21:04
0

Sadly there is no such "flag/property" which indicates that a given DynamoDb record is being processed already or not yet by a lambda function. You would need to design a custom solution for that.

One would would be to have other DynamoDb table, just to keep track of a current status of the deleted items. For example, when your lambda accepts a record it writes it in the table. Any other function or interested processes would have to check if the record exists or not in the table. The presence of the record would indicate that the item deleted is still being process.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • Can you please tell me suppose a record is deleted by TTL then it will be available in the streams then again a records is deleted by TTL then again it will be available in streams, So is there any way of identifying and processing both separately. – Reyan Chougle Jul 13 '21 at 13:01
  • @ReyanChougle As I wrote you need custom solution for that. – Marcin Jul 17 '21 at 07:32
0

To summarize: ask is to track the records which has already been TTL deleted. (I presume, you somehow wants to ensure not to reprocess them again)

Pretext:

  • As called out from @korgen, you would be able to verify userIdentity in stream record to identify the TTL deleted vs normal delte
  • You can relay on DDB on the fact that DDB stream contains unique events(not repeated or duplicate)
  • each stream record has a recordIdentifier(its a GUID) which helps track the location of record in stream. Alternatively you may use the HashKey + Range Key (+ record version, if present on DDB record) to build your own custom key

On top of my head, one solution that you can explore would work like this:

  1. While processing the TTL deleted record, persists in a secondary storage (uploading to s3).
  2. This way you can check this storage, if the record exist, then your lambda has already processed it.
mango
  • 548
  • 3
  • 8