0

We run daily jobs and store the data under the date folder in S3. The file contains a list of Identifier separated by Comma (Id1, Id2... Id100 etc).

Is there a way where we can add these values to DDB. I want this to happen everyday like a daily Job and update the DDB.

Is it also possible to just update the diff like

Day 1 has Id1, Id2.  -> Dynamo Updates to Id1, Id2
Day 2 has Id2, Id3.  -> Instead of deleting and adding all values, Can we override just the difference??
Vinny
  • 865
  • 1
  • 11
  • 25
  • 1
    Can you run things on a schedule in AWS? [Yes](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule-schedule.html). Can you make conditional updates to DynamoDB? [Yes](https://stackoverflow.com/questions/32777374/is-it-possible-to-do-a-conditional-put-or-update-in-dynamodb). – jarmod Apr 21 '23 at 18:25

1 Answers1

0

Depending on the amount of data you have to ingest daily would depend on how you achieve this.

If it's a small amount of daily data, using EventBridge and Lambda to run small ingestion jobs would work.

If it's several GB of data or more, then use AWS Glue.

Leeroy Hannigan
  • 11,409
  • 3
  • 14
  • 31
  • AWS Glue has only the connection from Dynamo -> S3, it does not have the reverse connection – Vinny Apr 21 '23 at 18:29
  • Of course it has. It uses both as source and sink. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html – Leeroy Hannigan Apr 21 '23 at 18:32