1

We have a requirement to delete DynamoDB items that are 3 days old so I tried default AWS CLI update query but the query doesn't take input value for TTL.

As per documentation, I'm following the below query to activate DynamoDB TTL (Time to Live) however when it is activated it defaults to one hour however I want a query that will take 3 days as the TTL value. How can I write the correct query? We are creating a script deployment so we won't be doing via UI console.

aws dynamodb update-time-to-live \
    --table-name MusicCollection \
    --time-to-live-specification Enabled=true,AttributeName=ttl

As shown in below image, items are not deleted passed the current time, I guess it deletes after an hour.

enter image description here

enter image description here

halfer
  • 19,824
  • 17
  • 99
  • 186
Mysterious288
  • 365
  • 5
  • 24

2 Answers2

3

DynamoDB Time to Live (TTL) only deals with you specifying a specific timestamp attribute as the TTL value & AWS deleting ('expiring') the record after the date and time of the timestamp has elapsed.

The aws dynamodb update-time-to-live only enables TTL on a specific attribute.

It does not set the TTL value for you.

The Simulated date and time option simply only simulates that specific point in time and then shows you a few items that would be expired. It's for your to check your TTL value. Again, it does not set it for you.

You will need to update your item(s) manually using put-item to specify a TTL value; there is no feature within DynamoDB to populate a TTL value.

Try using this as a starting point, if your script is in Bash:

EXP=`date -d '+3 days' +%s`
aws dynamodb put-item \
--table-name "MusicCollection" \
--item '{"id": {"N": "1"}, "ttl": {"N": "'$EXP'"}}'
Ermiya Eskandary
  • 15,323
  • 3
  • 31
  • 44
  • Got your point, so for every record, do I need to have this manual update and does it work? – Mysterious288 Nov 06 '22 at 11:06
  • @Mysterious288 Yes unfortunately, you will need to actually execute a put-item for each item. If you are looking to just get rid of xxxx items, it may be better to copy over the items you need with a TTL to a new table and then delete the old one. Also take a look at using [EMR with Hive](https://aws.amazon.com/blogs/database/backfilling-an-amazon-dynamodb-time-to-live-ttl-attribute-with-amazon-emr/) though it is much more complicated. – Ermiya Eskandary Nov 06 '22 at 11:19
  • With your point, I have started testing with the TTL value 1667733390 for an item, so far it's not yet been deleted, waiting for another 10mins. – Mysterious288 Nov 06 '22 at 11:21
  • Do not update items using put-item. Please use update-item instead. – Leeroy Hannigan Nov 06 '22 at 11:27
  • DynamoDB TTL is [eventually consistent](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/time-to-live-ttl-before-you-start.html#time-to-live-ttl-before-you-start-notes) - give it around 2 hours since you've just enabled it. – Ermiya Eskandary Nov 06 '22 at 11:28
  • 'Because TTL is meant to be a background process, the nature of the capacity used to expire and delete items via TTL is variable (but free of charge). TTL typically deletes expired items within 48 hours of expiration.' - https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html – Ermiya Eskandary Nov 06 '22 at 11:28
  • @LeeHannigan I'm just adding a new record with TTL value, I cannot update it would make no sense to do two operations for every record. – Mysterious288 Nov 06 '22 at 11:30
  • @ErmiyaEskandary It got deleted, so for every record, I have 3 days time in the future correct? Thanks a lot for your help and guidance. – Mysterious288 Nov 06 '22 at 11:31
  • Adding a new record with TTL value with put-item is fine. But if you were to update existing items use update-item as put-item is an overwrite method and could lead to data loss/inconsistencies. – Leeroy Hannigan Nov 06 '22 at 11:32
  • @LeeHannigan there are no existing records, this is a new requirement, so while creating a new record, I have to add 3 days time in the future as ttl correct? I understood it's usage now, thanks for your help. – Mysterious288 Nov 06 '22 at 11:34
  • That's correct, you can see in my answer that you need to set the date for TTL which is in the future. And be tolerant of the 48hr window which it may take to delete the item. – Leeroy Hannigan Nov 06 '22 at 11:41
2

TTL does not default to 1hour. TTL deletes items which exceed the current time.

For that reason if you want items to expire after 3 days then you need to set a TTL value on those items for 3 days in the future.

Bear in mind it can take up to 48 hours for TTL to evict the item, meaning the item will be removed between 3 and 5 days. To overcome this, you can add a FilterExpression to your read requests where you drop any items which are older than 3 days.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html

Leeroy Hannigan
  • 11,409
  • 3
  • 14
  • 31