Stack Overflow
Stack Overflow

Questions tagged [amazon-dynamodb-streams]

Use this tag for questions related to DynamoDB Streams, which provides a time ordered sequence of item level changes in any DynamoDB table. The changes are de-duplicated and stored for 24 hours. This capability enables you to extend the power of DynamoDB with cross-region replication, continuous analytics with Redshift integration, change notifications, and many other scenarios.

Useful links:

  1. Documentation
  2. Use cases
393 questions
11
votes
1 answer

cannot copy json - Dynamo db Streams to redshift

Following is the use case i am working on: I have configured enable Streams when creating DynamoDB with new and old Image.I have created a Kinesis Firehose delivery stream with Destination as Redshift(Intermediate s3). From Dynamodb my stream…
Prasanna Nandakumar
  • 4,295
  • 34
  • 63
11
votes
1 answer

DynamoDB Streams to S3

I am using Data Pipeline (DP) for daily backups of DynamoDB, however, I would like to do incremental backups of the data that is missed by DP runs (updates between DP runs). To accomplish that, I would like to use DynamoDB Streams + Lambda + S3 to…
user3293898
  • 271
  • 1
  • 3
  • 8
10
votes
2 answers

Ceating dynamodb table says "invalid One or more parameter values were invalid: Some index key attributes are not defined in AttributeDefinitions"

I am trying to create dynamo db table with python. Below is the script i have. I am trying to create one partition key and sort key and bunch of columns. What I tried: import boto3 dynamodb = boto3.resource('dynamodb') table =…
asp
  • 777
  • 3
  • 14
  • 33
10
votes
2 answers

AWS DynamoDB update item using multiple condition expression

I have a DynamoDB Table and I want a particular value to be updated using multiple condition expression. Can we do that? My code:- dynamodb = session.resource('dynamodb') table = dynamodb.Table('gold-images') response =…
Praveen Raghav
  • 163
  • 1
  • 2
  • 7
10
votes
1 answer

DynamoDB Local: Streams.getRecords returns TrimmedDataAccessException

I'm using DynamoDB local to run integration tests from nodejs (Javascript SDK). When I call getRecords I'm getting "TrimmedDataAccessException". If I run the same code against DynamoDB in AWS (not local) it works fine. Here's the steps: Run…
Mark J Miller
  • 4,751
  • 5
  • 44
  • 74
9
votes
3 answers

Increasing DynamoDB Stream + Lambda throughput

I have a DynamoDB Stream that triggers a Lambda function. I'm noticing that bursts of a thousand of writes to the DynamoDB table can take many minutes (longest I have seen is 30 minutes) to all be processed by Lambda. The average duration for each…
user1569339
  • 683
  • 8
  • 20
9
votes
7 answers

AmazonServiceException: Supplied AttributeValue is empty, must contain exactly one of the supported datatypes

I am trying to import data from dynamodb console interface, but unable to get success. Data…
rahulb
  • 970
  • 4
  • 12
  • 24
8
votes
1 answer

Filtering DynamoDB Streams events for a specific field change

I'm interested in value changes of a specific (numeric) field/column (say "count") of a DynamoDB table. I know that I could write a Lambda that, once triggered, compares the new image with the old one and then decides whether it is the case of…
cionzo
  • 428
  • 1
  • 5
  • 12
8
votes
2 answers

Does AWS Lambda process DynamoDB stream events strictly in order?

I'm in the process of writing a Lambda function that processes items from a DynamoDB stream. I thought part of the point behind Lambda was that if I have a large burst of events, it'll spin up enough instances to get through them concurrently,…
Laurence Gonsalves
  • 137,896
  • 35
  • 246
  • 299
8
votes
3 answers

How "Real-Time" DynamoDB stream is?

We are experimenting with a new serverless solution where external provider writes to DynamoDB, DynamoDB Stream reacts to a new write event, and triggers AWS Lambda function which propagates changes down the road? So far it works well, however,…
inside
  • 3,047
  • 10
  • 49
  • 75
7
votes
2 answers

How to add pre-existing data from DynamoDB to Elasticsearch?

I set up Elasticsearch Service and DynamoDb stream as described in this blog post. Now I need to add pre-existing data from DynamoDB to Elasticsearch. I saw "Indexing pre-existing content" part of article but I dont know what to do with that Python…
A. Bimer
  • 73
  • 1
  • 4
6
votes
1 answer

AWS Lambda: can one Lambda function have Event Sources of both Kinesis and DynamoDB Streams?

Can one AWS Lambda function have two Event Sources both of one Kinesis stream and one DynamoDB Stream? I've looked but I haven't found any documentation which says that I can or cannot have different types of event sources for the same AWS Lambda…
Alex Q
  • 3,080
  • 2
  • 27
  • 29
6
votes
2 answers

Set Validation rules for AWS DynamoDB table

Actually I want to implement validations on AWS DynamoDB table items, which should prevent records to insert/update if rules break for an item fields. Is it possible? Or Can we create a trigger lambda for dynamoDB table, that trigger before…
Sudhanshu
  • 121
  • 1
  • 2
  • 7
5
votes
1 answer

Use Kinesis Client Library v2.x with DynamoDB Stream Kinesis Adapter

I'm building a Java application to consume events from a DynamoDB Stream using the KCL library. The example on AWS docs uses this library: com.amazonaws:dynamodb-streams-kinesis-adapter:1.4.0 Which depends…
dipanda
  • 760
  • 1
  • 11
  • 24
5
votes
1 answer

Can DynamoDB stream see an uncomitted transaction?

I have a DynamoDB table where I am using transactional writes (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transactions.html). The transaction consists of 2 puts. Let's say the first put succeeds and the second fails. In this…
Nitesh Kumar
  • 789
  • 1
  • 5
  • 11
1
2
3
…
26 27