1

We are currently designing a dynamodb table to store certain file attributes. There are 2 main columns

  1. Date:- This contains the date in YYMMDD format for ex:-20190618
  2. FileName:- xxxxxxxxxxx.json

Currently the partition key is Date and sort key is FileName. We expect about 500000 files with distinct file names on each day (this can increase over period of time) . The File names will repeated same each day i.e. a typical schema is as shown below

Date FileName 20190617 abcd.json 20190618 abcd.json

We have a series of queries that is based on Date and a dynamodb trigger. The queries are working great. Currently what we are observing is that the number of concurrent lambda executions are limited to 2, since we are partition by date. While trying to improve the concurrency of lambda we came across 2 solutions

1) Referring the following link (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-sharding.html) , one idea is add a fixed number of random suffixes for Date Field i.e (20190617.1 to 20190617.500) to split the data in to 500 partitions with 1000 records each . This would ensure an amount of concurrency and also there will be minimal changed to query

2) Second option is to change partitioning of table as follows Partition Key :- FileName and SortKey :- Date. This will result in about 500000 partitions , (which can increase) . For querying by date we will need to add a GSI, but we will achieve more concurrency in Lambda

we have not created a table with 500000 partitions (which can increase). Any body has such experience... If so please comment

Any help is appreciated

Sabarish Sathasivan
  • 1,196
  • 2
  • 19
  • 42
  • Why do you think it would create 500000 partitions? – NoSQLKnowHow Jun 19 '19 at 03:58
  • For a single day there could be 500000 files will have different names – Sabarish Sathasivan Jun 19 '19 at 04:07
  • 1
    That is not how it works. Partitions are split and expanded based on partition size (10GB) or on throughput. e.g. Each partition can currently support 3000 IOPS. A read is 1 IOP and a write is worth 3 IOPS. If you go over that 3000 IOPS any partition or you provision more capacity than the current partitions can handle, then behind the scenes DynamoDB will create the necessary partitions to handle that load. – NoSQLKnowHow Jun 19 '19 at 18:48

2 Answers2

2

You seem to be under the mistaken impression that there's a one to one correspondence between partition keys and partitions.

This is not the case.

The number of partitions is driven by table size and through-put. The partition key is hashed by DDB and the data is stored in a particular partition.

You could have 100k partition keys and only a single partition.

If you're pushing the limits of DDB, then yeah you might end up with only a single partition key in a partition...but that's not typical.

The DDB Whitepaper provides some details into how DDB works...

Charles
  • 21,637
  • 1
  • 20
  • 44
  • Thanks a lot. This document along with the following link - https://shinesolutions.com/2016/06/27/a-deep-dive-into-dynamodb-partitions/ helped clear a lot of doubts – Sabarish Sathasivan Jun 19 '19 at 22:39
1

Partitioning by file name doesn’t make a lot of sense if your access pattern is to query by date.

Instead, the idea of increasing the number of partitions for each date by adding a suffix seems fine. But rather than adding a random suffix, you might consider adding a stable suffix based on the name of the file:

You could use the first letter of the file name, to get about 30 partitions - assuming the file names are random. The only trouble is some letter might be more common than others giving skewed subpartitions

Or, you could take a hash of the file name and use that as the suffix for the partition key. The hash function could be a relatively simple hash function that produces a target numeric value corresponding to the number of sub partitions you would like to have for each date.

If you end up with about 10000-50000 items per partition it would probably be great.

Hope this helps

Mike Dinescu
  • 54,171
  • 16
  • 118
  • 151
  • Thanks for your comments. It helps... One question is a dynamodb table with FileName partition key a good design.On the first day , there will be one record per partition , however during course of time the data will grow – Sabarish Sathasivan Jun 19 '19 at 12:54
  • 1
    If you use FileName as the partition key you loose the ability to query by date. It only makes sense if your access pattern is primarily based on FileName. So the answer is - it depends.. – Mike Dinescu Jun 19 '19 at 16:01