How to wire a DynamoDB stream to a kinesis stream?

Question

I was assuming I

create a table and enable stream and I now have an ARN
create a kinesis stream
configure somewhere to tell the dynamoDb stream to write to kinesis stream

I was looking at working with https://github.com/harlow/kinesis-consumer but this reads from kinesis or can I use the ARN and use it to read right from the dynamoDB stream?

The more I look, the more I seem to think, I have to write a lambda to read dynamoDB and write to kinesis. Is that correct?

thanks

Folger Fonseca · Answer 1 · 2019-09-15T07:39:07.207

0

Hey can you provide a bit more of information about your target setup? do you plan to have some sort of ETL process for your dynamoDB table? AFAIK when you bound a kinesis stream to a dynamodb table, everytime you add, remove or update rows on the dynamodb a new event will be publish in the associated kinesis stream which you can consume from and use the event in whatever way you want.

maybe worth checking this one: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.Walkthrough.html

edited Sep 15 '19 at 07:39

answered Sep 15 '19 at 07:29

Folger Fonseca

183
1
1
11

I didn't -1 you. I hate people that do that. Anyways @Folger Fonseca, dynamoDB streams is NOT a kinesis stream. In fact, they make an adapter on top of the dynamoDB stream api so you can use a kinesis api on top. kinesis stream has enhanced fanout while dynamoDB stream does not. – Dean Hiller Sep 15 '19 at 14:51
you are right, we use this approach in my company to populate our DWH, but I thought the stream will behave like any other normal kinesis stream, in such a case I think there won't be another way than to have a lambda to publish to your desire stream. Thanks @Dean – Folger Fonseca Sep 15 '19 at 20:20
Thats pretty cool! Does it still maintain the order of the records in Kinesis Data Streams as in the DynamoDB Streams? – dashuser Jan 03 '20 at 19:37

score 0 · Answer 2 · answered Nov 18 '22 at 10:53

DynamoDB now support Kinesis Data Streams natively:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html

You can choose either DynamoDB Streams or Kinesis Data Streams for your Change Data Capture (CDC).

Properties	Kinesis Data Streams for DynamoDB	DynamoDB Streams
Data retention	Up to 1 year.	24 hours.
Kinesis Client Library (KCL) support	Supports KCL versions 1.X and 2.X.	Supports KCL version 1.X.
Number of consumers	Up to 5 simultaneous consumers per shard, or up to 20 simultaneous consumers per shard with enhanced fan-out.	Up to 2 simultaneous consumers per shard.
Throughput quotas	Unlimited.	Subject to throughput quotas by DynamoDB table and AWS Region.
Record delivery model	Pull model over HTTP using GetRecords and with enhanced fan-out, Kinesis Data Streams pushes the records over HTTP/2 by using SubscribeToShard.	Pull model over HTTP using GetRecords.
Ordering of records	The timestamp attribute on each stream record can be used to identify the actual order in which changes occurred in the DynamoDB table.	For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.
Duplicate records	Duplicate records might occasionally appear in the stream.	No duplicate records appear in the stream.
Stream processing options	Process stream records using AWS Lambda, Kinesis Data Analytics, Kinesis data firehose , or AWS Glue streaming ETL.	Process stream records using AWS Lambda or DynamoDB Streams Kinesis adapter.
Durability level	Availability zones to provide automatic failover without interruption.	Availability zones to provide automatic failover without interruption.

score 0 · Answer 3 · edited Nov 22 '22 at 00:33

You can use Amazon Kinesis Data Streams to capture changes to Amazon DynamoDB. According to the AWS documentation:

Kinesis Data Streams captures item-level modifications in any DynamoDB table and replicates them to a Kinesis data stream. Your applications can access this stream and view item-level changes in near-real time. You can continuously capture and store terabytes of data per hour. You can take advantage of longer data retention time—and with enhanced fan-out capability, you can simultaneously reach two or more downstream applications. Other benefits include additional audit and security transparency.

Also You can enable streaming to Kinesis from your DynamoDB table.

How to wire a DynamoDB stream to a kinesis stream?

3 Answers3