DynamoDB scan shows fewer items than DynamoDB console

Question

Why does a scan with nodejs shows just 3 result, while the dynamodb admin tool shows 9

var params = {
    TableName: process.env.DYNAMODB_TABLE_LIGHTHOUSE,
    FilterExpression: '#blickArticleId = :valblickArticleId AND #firstFolder = :valfirstFolder',
    ExpressionAttributeNames: {
        '#blickArticleId': 'blickArticleId',
        '#firstFolder': 'firstFolder'
    },
    ExpressionAttributeValues: {
        ':valblickArticleId': 'null',
        ':valfirstFolder': 'null'
    },
  };

  const queryResponse = await dynamoDb.scan(params).promise()

isnt that the same

Is LastEvaluatedKey present and not empty in the (mis-named) queryResponse? — jarmod, May 07 '19 at 17:24
"Count":3,"ScannedCount":29,"LastEvaluatedKey":{"id":"db21e190-70d0-11e9-8e71-c142a0fbc609"}} looks good — Tobi, May 07 '19 at 18:07
OK, so you need to scan for the remaining items. The presence of a non-null LastEvaluatedKey indicates that DynamoDB results are paginated. The AWS DynamoDB console is presumably doing the pagination for you. See https://stackoverflow.com/questions/44589967/how-to-fetch-scan-all-items-from-aws-dynamodb-using-node-js — jarmod, May 07 '19 at 18:13

score 2 · Answer 1 · answered May 08 '19 at 11:44

Are you sure , your scanned content is not more than 1MB ?.

If the total number of scanned items exceeds the maximum data set size limit of 1 MB, the scan stops and results are returned to the user with a LastEvaluatedKey

Then you can scan the remaining items by using LastEvaluatedKey.

score 1 · Accepted Answer · answered May 07 '19 at 18:35

as jarmod mentioned pagination is the solution:

const getLalalalalal = async () => {
  var params = {
    TableName: process.env.DYNAMODB_TABLE_LIGHTHOUSE,
    FilterExpression: '#blickArticleId = :valblickArticleId AND #firstFolder = :valfirstFolder',
    ExpressionAttributeNames: {
        '#blickArticleId': 'blickArticleId',
        '#firstFolder': 'firstFolder'
    },
    ExpressionAttributeValues: {
        ':valblickArticleId': 'null',
        ':valfirstFolder': 'null'
    },
  };

  return await scanTable(params)
}

const scanTable = async (params) => {
  let scanResults = [];
  let items;
  do {
      items = await dynamoDb.scan(params).promise();
      items.Items.forEach((item) => scanResults.push(item));
      params.ExclusiveStartKey = items.LastEvaluatedKey;
  } while(typeof items.LastEvaluatedKey != "undefined");

  return scanResults;
};

score 1 · Answer 3 · answered May 07 '19 at 19:58

Your code needs to scan for the remaining items. The presence of a non-null LastEvaluatedKey indicates that DynamoDB results are paginated. The AWS DynamoDB console is presumably doing the pagination for you.

Here is an example of code to fetch all items if paginated.

DynamoDB scan shows fewer items than DynamoDB console

3 Answers3