AWS DMS CDC task does not detect column name and type changes

Question

I have created a CDC task that captures changes in a source PostgreSQL schema and writes them in Parquet format into a target S3 bucket. The task captures the inserts, updates and deletes correctly but fails to capture column name and type changes in the source.

When I change a column name or type of a table in the source and insert new rows to the table, the resulting Parquet file uses the old column name and type.

Is there a specific configuration I am missing? or it is not possible to achieve the desired outcome from this task in DMS?

if you change column at source and DMS will pick automatically from source and update at destination. check your DMS setting. you no need to do manually adding column at destination. — GRVPrasad, Feb 24 '20 at 10:56
Under Table Statistics --- Validate Again you have to do it. — GRVPrasad, Feb 24 '20 at 11:20
according to the docs only a limited number of DDL commands are supported: TRUNCATE TABLE, DROP TABLE, and CREATE TABLE https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html — RudyVerboven, May 11 '20 at 13:10

score 1 · Answer 1 · answered Feb 24 '20 at 11:21

1

if you change column at source and DMS will pick automatically from source and update at destination. check your DMS setting. you no need to do manually adding column at destination

answered Feb 24 '20 at 11:21

GRVPrasad

1,228
1
9
24

score 0 · Answer 2 · answered Apr 17 '20 at 13:29

Make sure you have the HandleSourceTableAltered parameter set to true in the task settings.[1] (The setting applies when the target metadata parameter BatchApplyEnabled is set to either true or false.)

Same goes for HandleSourceTableDropped or HandleSourceTableTruncated if this is relevant in your case.

Obviously, previously replicated Parquet files on S3 will not change to reflect this DDL change on the source.

[1] https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.DDLHandling.html

AWS DMS CDC task does not detect column name and type changes

2 Answers2