0

My requirement is to stream only INSERTs on a specific table in my db to a Kinesis data stream.

I have configured this pipeline in my AWS environment:

RDS Postgres 13 -> DMS (Database Migration Service) -> KDS (Kinesis Data Stream)

This setup works correctly but it processes all changes, even UPDATEs and DELETEs, on my source table.

What I've tried:

  • Looking for config options in the Postgres logical decoding plugin. DMS uses the test_decoding PG plugin which does not accept options to include/exclude data changes by operation type.
  • Looking at the DMS selection and filtering rules. Still didn't see anything that might help.

Of course I could simply ignore records originated from non-INSERT operations in my Kinesis consumer, but this doesn't look like a cost-efficient implementation.

Is there any way to meet my requirements using these AWS services (RDS -> DMS -> Kinesis)?

raxaghi
  • 3
  • 2

1 Answers1

0

Well DMS does not have this capability .

If you want only INSERT to be send to Kinesis in that case you can have a lambda function on every INSERT of RDS . Lambda function can be configured as trigger for INSERT .

You can invoke lambda only for INSERT and write to Kinesis directly .

Cost wise also this will be less . In DMS you are paying for Replication instance even when not in use .

For detailed reference Stream changes from Amazon RDS for PostgreSQL using Amazon Kinesis Data Streams and AWS Lambda

Sudarshan kumar
  • 1,503
  • 4
  • 36
  • 83