0

I have a DynamoDB transaction which appends > 1 records at any time in a single DynamoDB table using transactWrite. For example, in a single transaction, I can append A, B, and C records. Note that in my case, the operations are always append only (inserts only).

The records are then passed over to DynamoDB stream and to a lambda for processing. However, sometimes, lambda receives the events out of order. I understand that behavior I think because from DynamoDB's point of view, all 3 events were written at the same timestamp. So, there is no ordering. But if these events are part of same batch, I can always reorder them in the lambda before processing.

However, that is where the problem is. Even though these records are written in single transaction, they don't always appear together in the same batch in the lambda. Sometimes, I receive C as the only event and then A, B arrive in a batch later on. I think that the behavior is somewhat reasonable. Is there a way to guarantee that I receive all the records written in a transaction in one single batch.

NoSQLKnowHow
  • 4,449
  • 23
  • 35
drcocoa
  • 1,155
  • 1
  • 14
  • 23

1 Answers1

2

Your items may be written in a single transaction, but each item could be in a separate stream shard. Streams have shards, therefore it is possible that each item each arrives at the same time, but each of items in the streams land on different stream shards. Ordering is by time and item, not overall keyspace and time: "For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.” It is possible to ensure ordered updates to each item, but if you need to have consistency across all updates in the keyspace then this would need to be designed on the reader side.

All that said, I wonder if there is an opportunity to denormalize these three items into one item on the base table and skip using TransactWriteItem altogether.

NoSQLKnowHow
  • 4,449
  • 23
  • 35
  • You are correct. I didn’t realize that until I ran some rests. I couldn’t pin point the exact reason why it happened but what you said about shards in a stream makes sense. In my case, I publish the events from dynamo stream to a kinesis stream. So to solve this issue, I introduced an intermediate “parking” stream where these events are sent. A lambda keeps track of event sequence number in a dynamodb table. If the event arrives out of order, it puts it back in parking stream. Otherwise, it forwards to the original stream. – drcocoa Jul 18 '20 at 17:34
  • In my case, I have a monotonically increasing unique sequence number for each event per entity id. So in the reordering lambda, I only track the entity id and its last sequence number. – drcocoa Jul 18 '20 at 17:38