Is there a way to GET all items in a global secondary index with REST API using aws api-gateway? I can only GET some

Question

I created a REST api using aws api-gateway and dynamodb without using aws-lambda (I wrote mapping templates for both the integration request and integration response instead of lambda) on a GET API method, POST http method and Scan action setting. I'm fetching from a global secondary index in dynamodb to make my scan smaller than the original table.

It's working well except I am only able to scan roughly 1,000 of my 7,500 items that I need to scan. I checked out paginating the json in an s3 bucket, but I really want to keep it simple with just the aws api-gateway and the dynamodb, if possible.

Is there a way to get all 7,500 of the items in my payload with some modification to my integration request and/or response mappings? If not, what do you suggest?

Below is the mapping code I'm using that works for a 1000 item json payload instead of the 7,500 that I would like to have:

Integration Request:

{ 
  "TableName": "TrailData",
  "IndexName": "trail-index"
}

Integration Response:

  #set($inputRoot = $input.path('$'))
  [
   #foreach($elem in $inputRoot.Items)
      { 
       "id":$elem.id.N,
       "trail_name":"$elem.trail_name.S",
       "challenge_rank":$elem.challenge_rank.N,
       "challenge_description":"$elem.challenge_description.S",
       "reliability_description":"$elem.reliability_description.S"
       }
       #if($foreach.hasNext),#end
       #end
  ]

Here is a screenshot of the GET method settings for my API: API Screenshot

I have already checked out this: stackoverflow question related topic, but I can't figure out how to apply it to my situation. I have put a lot of time into this.

I am aware of the 1MB query limit for dynamodb, but the limited data I am returning is only 142KB.

I appreciate any help or suggestions. I am new to this. Thank you!

Answer 1

This limitation is not related to Dynamo Scan but VTL within Response Template #foreach is restricted to 1000 iterations Here is the issue.

We can also confirm this, by simply removing the #foreach(or entire response template), we should see all(1MB) the records back (but not well formatted).

Easiest solution is pass the request parameters to restrict only necessary attributes from Dynamo table

{
    "TableName":"ana-qa-linkshare",
    "Limit":2000,
    "ProjectionExpression":"challenge_rank,reliability_description,trail_name"
}

However, we can avoid doing a single loop that goes over 1000 with multiple foreach loops, but going to get little complex with in template, instead, we could use lambda. But here is how it might look like.

#set($inputRoot = $input.path('$'))
#set($maxRec = 500)
#set($totalLoops = $inputRoot.Count / $maxRec )
#set($outerArray = [0..$totalLoops])
 #set($innerArray = [0..$maxRec])
 {
  [
  #foreach($outer in $outerArray)
  #foreach($inner in $innerArray)
      { 
    grab the element with $inputRoot.Items.get(..index)
      and Build JSON here.
       }
    #end   
    #end   
  ]
}