Streaming data from AWS DMS to AWS MSK using lambda function

Question

So i am new to Kafka and MSK but trying to explore thing server less on AWS cloud . My use case AWS DMS-->KAFKA(MSK)-->ElastciSearch/DynamoDB and S3. i am looking to do this using lambda function. But recently i came to know end to end can not be or difficult to fit into lambda function .

So is there any way to implement this using Kafka connect with lambda function so that no EC2 would be required for this ?

Basically i am looking for Kafka connect in serverless way.

Kinesis is ruled out for this option and not in this scope .

Please suggest something with Kafka

Please post a new question rather than invalidate existing answers — OneCricketeer, Dec 23 '19 at 21:58

score 0 · Answer 1 · answered Dec 21 '19 at 02:40

0

Kafka Connect must run in EC2 or EKS/ECS, as it's a long running service, not something that starts and stops like Lambda

I'm not sure what DMS completely provides, but I assume you could do the same thing using Debezium to replicate one database into another via Kafka

answered Dec 21 '19 at 02:40

OneCricketeer

179,855
19
132
245

Can please share something how can i set up this on EC2 .Any related docs will work . – Sudarshan kumar Dec 23 '19 at 09:26
Such as https://debezium.io? Or http://www.carbonrider.com/2019/11/16/change-data-capture-with-apache-kafka-postgresql-kafka-connect-and-debezium/ – OneCricketeer Dec 23 '19 at 21:56

Streaming data from AWS DMS to AWS MSK using lambda function

1 Answers1