0

I am trying to fetch data from AWS Dynamo DB scan with contains FilterExpression .I am using the following param but receiving empty response as:

{"Items":[],"Count":0,"ScannedCount":792,"LastEvaluatedKey":{"ID":"b123456789"}}"}

where i want to filter based on string from array ex: orange from fruits:['orange', 'apple', 'banana'];

Param for scan operation:

params = { TableName: table, FilterExpression: "contains (fruits, :L)", ExpressionAttributeValues: { ":L": { S :"orange" } } }

but when i use the same as command of AWS CLI. I am getting data in following format :

{"Items": [{"fruits": {"L": [{"S": "orange"}]}},{}]}

Can you please let me know what is wrong with the param i am using.

vrk
  • 5
  • 3

1 Answers1

0

To understand what is happening here, you need to understand how DynamoDB filters data and paginates results. It happens in this order:

  1. Read items from the table
  2. Apply Filter
  3. Return Results

DynamoDB query and scan operations return up to 1MB of data at a time. Anything beyond that will be paginated. You know your results are being paginated if DynamoDB returns a LastEvaluatedKey element in your response.

Filters apply after the 1MB limit. This is the critical step that often catches people off-guard. In your situation, the following is happening:

  1. You execute a scan operation that reads 1MB of data from the table.
  2. You apply a filter to the 1MB response, which results in all of the records in the first step to be eliminated from the response.
  3. DDB returns the remaining items with a LastEvaluatedKey element, which indicates there is more data to search.

In other words, your filter isn't applying to the entire table. It's applying to 1MB of the table at a time. In order to get the results you are looking for, you are going to need to execute the scan operation repeatedly until you reach the last "page" of the table.

Seth Geoghegan
  • 5,372
  • 2
  • 8
  • 23
  • Hi Seth, But i am not getting Items in any of the 1MB series. – vrk Nov 24 '20 at 05:31
  • I'm not sure I understand the problem you are facing. Your initial question says you are getting empty results but later says you are getting results but in a different format. Can you clarify the problem you're facing? It would be helpful to see what code or CLI commands you are executing. – Seth Geoghegan Nov 24 '20 at 16:13