DynamoDB with boto3 - limit acts as page size

Question

According to the boto3 docs, the limit argument in query allows you to to limit the number of evaluated objects in your DynamoDB table/GSI.

However, LastEvaluatedKey isn't returned when the desired limit is reached and therefore a client that would like to limit the number of fetched results will fail to do so

consider the following code:

        while True:
            query_result = self._dynamodb_client.query(**query_kwargs)
            for dynamodb_formatted_item in query_result["Items"]:
                yield self._convert_dict_from_dynamodb_key_names(
                    from_dynamodb_formatted_dict_to_dict(dynamodb_formatted_item)
                )

            if "LastEvaluatedKey" not in query_result:
                return

Am I missing something here ? Is this a bug in the Boto library ?

score 9 · Accepted Answer · answered Jun 05 '19 at 21:30

Your example code is missing the critical part where LastEvaluatedKey is fed back into the query, as an ExclusiveStartKey parameter! So you are retrying the same query in a loop, rather than continuing where the previous query stopped.

For example, here is working code (I generated an array, it's not a cool generator like you did ;-)):

def full_query(table, **kwargs):
    response = table.query(**kwargs)
    items = response['Items']
    while 'LastEvaluatedKey' in response:
        response = table.query(ExclusiveStartKey=response['LastEvaluatedKey'], **kwards)
        items.extend(response['Items'])
    return items

You can now run

full_query(Limit=37, KeyConditions={...})

And get all the results, fetched in batches of 37.

DynamoDB with boto3 - limit acts as page size

1 Answers1

Linked