2

I am trying to optimize my query to dynamodb to return all matching items. From the cli I can query the table and it returns all 303 items.

CLI Request:

aws dynamodb query --table-name my-table --index-name "myGsi" --key-condition-expression 'myId=:someId' --expression-attribute-values '{":someId":{"S":"86c10962-6444-4f09-8f6c-dc6b30abac25"}}' Response:

{
    "Items": [
         { ... },
         { ... }
     ],
    "Count": 303,
    "ScannedCount": 303,
    "ConsumedCapacity": null
}

The problem is that when I try to recreate that query in my Typescript application, I only get ~60 items in the response (not always consistent).

Code:

const params: DocumentClient.QueryInput = {
  TableName: 'my-table',
  IndexName: 'myGsi',
  KeyConditionExpression: 'myId = :someId',
  ExpressionAttributeValues: {
    ':someId': '86c10962-6444-4f09-8f6c-dc6b30abac25',
  }
};
const result: DocumentClient.QueryOutput = await dynamoClient.query(params).promise();

However, the result here is much smaller, and I need to paginate:

{
    "Items": [
         { ... },
         { ... }
     ],
    "Count": 59,
    "ScannedCount": 59,
    "LastEvaluatedKey": "xyz"
}

Any ideas on how to get all/more of the content in a single request from the SDK? Why would the results be different than my CLI request?

Brian Anderson
  • 621
  • 7
  • 22

1 Answers1

1

From the docs:

A single Query operation will read up to the maximum number of items set (if using the Limit parameter) or a maximum of 1 MB of data and then apply any filtering to the results using FilterExpression. If LastEvaluatedKey is present in the response, you will need to paginate the result set. For more information, see Paginating the Results in the Amazon DynamoDB Developer Guide.

Don't know if I can add something else to it. That's the expected behavior.

Any ideas on how to get all/more of the content in a single request from the SDK?

Look's like it's impossible with the original library. Unless you fork it and hack it. But there're 2 other options:

  • you can try to set the Limit to some insanely big value, and maybe it'll override the hardcoded 1 MB limit in the library. Don't know - haven't tried it.
  • or you can use aws dynamodb query through exec() (or some similar child_process's funcs) if you really need to.

In either case, be warned - it can cost you money. So maybe it's better to stick to pagination.

x00
  • 13,643
  • 3
  • 16
  • 40
  • Yes, I understood the docs, and what you mentioned is current. However, I still don't understand why the results from the CLI would be different. Calling a child_process would work, but it is such a hack! Thank you for your feedback! – Brian Anderson Oct 22 '21 at 14:37
  • I imagine that's exactly because it's a hack - seems that this CLI is supposed to be used manually, so AWS doesn't care about protecting against overcharge. But, as I said, if you won't use pagination it can **unexpectedly** lead to a huge quota usage. – x00 Oct 22 '21 at 15:10