3

I am currently trying to run a daily snapshot (I schedule it to run every day) of an RDS (Postgres) database using AWS DMS. My destination endpoint is S3 and I am using Full Load Replication. My goal is to write every snapshot into a separate date partition. For example, I would like to write today's snapshot into an S3 folder partition that looks like: database_schema_name/table_name/2021/11/06/13/LOAD00000001.csv.

However, it seems like I am unable to achieve the folder partitioning with full loads. Reading the docs on date-based folder partitioning (here):

You can enable date-based folder partitioning when you create an S3 target endpoint. You can enable it when you either migrate existing data and replicate ongoing changes (full load + CDC), or replicate data changes only (CDC only).

To my understanding of the docs date-based folder partitioning seems to be only available for CDC or CDC + full loads. I have also tried using it myself but without success.

To summarize, my goal is to be able to run full-loads via DMS and to place each day's load into a date-partitioned folder structure within S3. I am aware that I can write to S3 then place the written files into folder using Lambdas, but I was hoping to achieve that cleanly using DMS and without adding a further complexity to the system.

Reynold Rogers
  • 79
  • 5
  • If you're doing this to be able to restore a point in time, should be aware you can point in time recover with regular RDS snapshots. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.html – jordanm Nov 06 '21 at 17:40
  • Hi @jordanm. I updated my question to clarify my intent. I am trying to run it in an ETL fashion. So every day I schedule a job that creates a new snapshot. – Reynold Rogers Nov 07 '21 at 07:21

0 Answers0