0

I have a dynamo db table with following structure

partitionKey - userId+keyName
sortKey - keyName+itemId
itemData - any object
createdAt - long value
updatedAt - long value

In this table I want to save list of items lets say all unique eatable items found in a shop. As per the requirement I need to find out the count of items in a particular shop. As per my findings I came across three ways to do this

  1. Use Query to fetch count as per this link without explicitly saving count value.
  2. Use transactions while saving items and store/update count explicitly. [We want to add/remove multiple items in a single request]. And later get count using GetItem api.
  3. Use dynamo db streams to trigger SNS and eventually store explicit count in the same table/different table. And later get count using GetItem api.

Note

  • Latency is important here along with the cost.
  • You can assume this dynamo db table can have millions of items.
  • Eventual consistency is fine.

In my view 3rd option looks more efficient in terms of cost, latency. But want to know if my thoughts are correct

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197

1 Answers1

0

Using Dynamo streams to write aggregate data back to Dynamo is definitely the way to go!

This will of course be eventually consistent by its nature, as updating your item and waiting for the stream to update the aggregate are two different non-atomic operations.

A fourth option would be to have something like an ElasticSearch index updated (also by using streams), which allows you to do arbitrary ad-hoc queries.

If you need consistency for your aggregates, you have to use transactions for this, with all the limitations imposed.

August Lilleaas
  • 54,010
  • 13
  • 102
  • 111
  • Can you also comment on problems with using Query approach? – Shubham Agarwal Bhewanewala Aug 18 '21 at 18:40
  • I read the link, and don't really have anything to add to all the information in that SO post :) Queries are fundamentally limited to how much data they can read in storage before they terminate, and scans obviously don't scale, so if you expect arbitrarily sized data sets, you have to aggregate in dynamo or replicate data to some other database with counting built-in – August Lilleaas Aug 19 '21 at 12:09