Is there a way to efficiently fetch all the results from a medium sized DynamoDB?

Question

I'm using boto3 with python, but I belive the problem and logic should be universal across all languages.

I know that table.scan() should in theory return all the records, but in fact, they scan() results are limited to 1MB in size. It's recommended to create a while loop based on LastEvaluatedKey, but that also doesn't give me all the results (15200 instead of 16000), code here:

dynamodb = boto3.resource('dynamodb', region_name='eu-west-2')
table = dynamodb.Table(dBTable)
response = table.scan()
print("item_count:", table.item_count)
print("response1:", response["Count"])

items=[]
while 'LastEvaluatedKey' in response and response['LastEvaluatedKey'] != "":  
    print("response:", response["Count"])
    items+=response["Items"]
    response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])

How can I fetch all the records reliably?

E.J. Brennan · Answer 1 · 2020-11-16T20:11:39.673

2

It's recommended to create a while loop based on LastEvaluatedKey, but that also doesn't give me all the results (15200 instead of 16000).

Are you sure about that? My guess would be you have something else going on. I use boto3, and LastEvaludatedKey in a loop in production code that runs every day, and have never encountered a case where not all the rows were returned - not saying it's impossible, but I would first make sure you code is right.

Edit, this code works:

import boto3

from boto3.dynamodb.conditions import Key, Attr

dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
table = dynamodb.Table('DeadLetterQueue')
response = table.scan()
print("item_count:", table.item_count)

items=response["Items"]
while 'LastEvaluatedKey' in response and response['LastEvaluatedKey'] != "":  
    response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
    items.extend(response["Items"])


print (len(items))

edited Nov 16 '20 at 20:11

answered Nov 16 '20 at 19:33

E.J. Brennan

45,870
7
88
116

I've added code to my question, would you mind having a look? I'm running this on lambda - if it matters – madej Nov 16 '20 at 19:48
I have added working code for you, slightly different than yours, but you should be able to adapt it. – E.J. Brennan Nov 16 '20 at 20:12
thanks, that worked! it was all about the order of lines – madej Nov 16 '20 at 20:38

score 0 · Accepted Answer · answered Nov 16 '20 at 20:27

The problem you are facing is not related to the DynamoDB scan operation. It's related to your code. The last scan operation is not appending to the items array.

Following is your code with slight modifications -

dynamodb = boto3.resource('dynamodb', region_name='eu-west-2')
table = dynamodb.Table(dBTable)
response = table.scan()
print("item_count:", table.item_count)
print("response1:", response["Count"])

items=response["Items"] // Changed HERE.
while 'LastEvaluatedKey' in response and response['LastEvaluatedKey'] != "":  
    response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
    print("response:", response["Count"])
    items+=response["Items"] // Shifted this line HERE

Is there a way to efficiently fetch all the results from a medium sized DynamoDB?

2 Answers2