2

I created a REST api using aws api-gateway and dynamodb without using aws-lambda (I wrote mapping templates for both the integration request and integration response instead of lambda) on a GET API method, POST http method and Scan action setting. I'm fetching from a global secondary index in dynamodb to make my scan smaller than the original table.

It's working well except I am only able to scan roughly 1,000 of my 7,500 items that I need to scan. I checked out paginating the json in an s3 bucket, but I really want to keep it simple with just the aws api-gateway and the dynamodb, if possible.

Is there a way to get all 7,500 of the items in my payload with some modification to my integration request and/or response mappings? If not, what do you suggest?

Below is the mapping code I'm using that works for a 1000 item json payload instead of the 7,500 that I would like to have:

Integration Request:

{ 
  "TableName": "TrailData",
  "IndexName": "trail-index"
}

Integration Response:

  #set($inputRoot = $input.path('$'))
  [
   #foreach($elem in $inputRoot.Items)
      { 
       "id":$elem.id.N,
       "trail_name":"$elem.trail_name.S",
       "challenge_rank":$elem.challenge_rank.N,
       "challenge_description":"$elem.challenge_description.S",
       "reliability_description":"$elem.reliability_description.S"
       }
       #if($foreach.hasNext),#end
       #end
  ]

Here is a screenshot of the GET method settings for my API: API Screenshot

I have already checked out this: stackoverflow question related topic, but I can't figure out how to apply it to my situation. I have put a lot of time into this.

I am aware of the 1MB query limit for dynamodb, but the limited data I am returning is only 142KB.

I appreciate any help or suggestions. I am new to this. Thank you!

jelhan
  • 6,149
  • 1
  • 19
  • 35

1 Answers1

1

This limitation is not related to Dynamo Scan but VTL within Response Template #foreach is restricted to 1000 iterations Here is the issue.

We can also confirm this, by simply removing the #foreach(or entire response template), we should see all(1MB) the records back (but not well formatted).

Easiest solution is pass the request parameters to restrict only necessary attributes from Dynamo table

{
    "TableName":"ana-qa-linkshare",
    "Limit":2000,
    "ProjectionExpression":"challenge_rank,reliability_description,trail_name"
}

However, we can avoid doing a single loop that goes over 1000 with multiple foreach loops, but going to get little complex with in template, instead, we could use lambda. But here is how it might look like.

#set($inputRoot = $input.path('$'))
#set($maxRec = 500)
#set($totalLoops = $inputRoot.Count / $maxRec )
#set($outerArray = [0..$totalLoops])
 #set($innerArray = [0..$maxRec])
 {
  [
  #foreach($outer in $outerArray)
  #foreach($inner in $innerArray)
      { 
    grab the element with $inputRoot.Items.get(..index)
      and Build JSON here.
       }
    #end   
    #end   
  ]
}
Balu Vyamajala
  • 9,287
  • 1
  • 20
  • 42
  • output from Dynamo is not the best readable format for client with all `S/N` I will keep looking if there is an alternative for `foreach` – Balu Vyamajala Jan 17 '21 at 04:04
  • I just read your link about the 1,000 iteration issue. Funny, I also noticed that it renders invalid json to boot with the trailing comma. Using a counter is such a hack, but can be done...but, what about the "rest" of the data? Thanks for helping me look further. I'm still really stuck on this. One reason why the request parameter thing may not work is because the json scan will be behind an autocomplete field. Or, I am not aware of how to do that. – Marci Sprouse Jan 17 '21 at 05:35
  • without any response template, I am getting full data from Dynamo back to client though? are you getting invalid json with no response template? – Balu Vyamajala Jan 17 '21 at 05:39
  • I am thrilled to tell you that it worked! I wonder why I couldn't figure that out? THANK YOU so much! – Marci Sprouse Jan 17 '21 at 05:41
  • I updated answer with alternate approach that might work to avoid foreach over 1000! – Balu Vyamajala Jan 17 '21 at 05:43