0

I have multiple files present in different buckets in S3. I need to move these files to Amazon Aurora PostgreSQL every day on a schedule. Every day I will get a new file and, based on the data, insert or update will happen. I was using Glue for insert but with upsert Glue doesn't seem to be the right option. Is there a better way to handle this? I saw Load command from S3 to RDS will solve the issue but didn't get enough details on it. Any recommendations please?

jarmod
  • 71,565
  • 16
  • 115
  • 122
Hary S
  • 140
  • 1
  • 2
  • 13
  • There is documentation available for migrating data from S3 to Aurora (https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Migrating.html). Did you read that and try anything? Which option did you try and what failed? – jarmod Jun 14 '20 at 15:56
  • Hi, Thanks for sharing this link! The link you shared doesnt have the option to schedule the job daily and also it has covered details about import. Can you please tell me if upsert works? – Hary S Jun 14 '20 at 16:36

1 Answers1

0

You can trigger a Lambda function from S3 events, that could then process the file(s) and inject them into Aurora. Alternatively you can create a cron-type function that will run daily on whatever schedule you define.

https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

Ville Rinne
  • 801
  • 1
  • 7
  • 15