I have an AWS DMS database migration task running Full Load and Replication. The source is RDS (Aurora PostgreSQL) and I have pglogical configured.
The task ran the initial load just fine, and replication was working until we changed the name of one table in the source database, and dropped another table. At this point the task started failing. Here's a subset of the error messages:
Matched table found, owner is 'public' name is 'organization_tos_accepts'
Matched table found, owner is 'public' name is 'side_by_side_stripe_responses'
relation cannot be null
Unable to add table 'public.side_by_side_stripe_responses' to replication set...
Failed to process - ' select case when exists (select * from pglogical.tables where nspname = 'public' and relname = 'side_by_side_stripe_responses'
Error occurred during Get DDL type for 'public.organization_tos_accept'...
Parsing DDL data has failed.
side_by_side_stripe_responses
is the table that was dropped, and organization_tos_accept
was renamed to organization_tos_accepts
It appears that pglogical is still showing these tables as needing to be replicated, but when DMS tries to get information about them, it fails.
What do I need to do to get the DMS task to run successfully? I don't need the table-level statements (alter table, drop table, etc.) to replicate - I'm consuming the output with a lambda and only care about record inserts, updates, and deletes. I'd like for DMS to just ignore these two tables altogether, as they no longer exist. Note that I have tried modifying the task with selection rules to exclude those two tables, but I still see the same errors.
Terraform for dms task:
resource "aws_dms_replication_task" "warehouse-prod-replication-task" {
migration_type = "full-load-and-cdc"
replication_instance_arn = aws_dms_replication_instance.production.replication_instance_arn
replication_task_id = "warehouse-prod-replication-task"
table_mappings = "{\"rules\":[{\"rule-type\":\"selection\",\"rule-id\":\"1\",\"rule-name\":\"warehouse-prod-replication-task\",\"object-locator\":{\"schema-name\":\"public\",\"table-name\":\"%\"},\"rule-action\":\"include\",\"filters\":[]}]}"
source_endpoint_arn
= aws_dms_endpoint.dms_source_endpoint.endpoint_arn
target_endpoint_arn
= aws_dms_endpoint.dms_target_endpoint.endpoint_arn
start_replication_task = "true"
replication_task_settings = jsonencode(
{
Logging = {
EnableLogging = true
}
}
)
}